Apr 22 16:25:16 fir-io2-s2 kernel: Lustre: Skipped 602 previous similar messages
Apr 22 16:37:26 fir-io2-s2 kernel: Lustre: fir-OST0011: Connection restored to 70ef6e44-4b1c-8255-8845-5cc3a5ab05ff (at 10.9.107.64@o2ib4)
Apr 22 16:37:26 fir-io2-s2 kernel: Lustre: Skipped 131 previous similar messages
Apr 22 16:40:53 fir-io2-s2 kernel: Lustre: fir-OST000f: haven't heard from client 587ea0b9-e7ff-8058-1bc0-a978b603fc9d (at 10.8.27.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91120270a400, cur 1555976453 expire 1555976303 last 1555976226
Apr 22 16:40:53 fir-io2-s2 kernel: Lustre: Skipped 29 previous similar messages
Apr 22 16:44:30 fir-io2-s2 kernel: Lustre: fir-OST0015: Client 36db31da-4b3a-423e-953b-75d4eb2a9f29 (at 10.8.30.21@o2ib6) reconnecting
Apr 22 16:44:30 fir-io2-s2 kernel: Lustre: Skipped 4 previous similar messages
Apr 22 16:44:30 fir-io2-s2 kernel: Lustre: fir-OST0015: Client f653589b-eefb-abf7-a1b5-4c7dd788fc78 (at 10.8.7.16@o2ib6) reconnecting
Apr 22 16:44:30 fir-io2-s2 kernel: Lustre: Skipped 1 previous similar message
Apr 22 16:44:35 fir-io2-s2 kernel: Lustre: fir-OST0011: Client 27b0e7ba-9e52-b4a7-2089-53629fd5cc62 (at 10.8.10.24@o2ib6) reconnecting
Apr 22 16:44:35 fir-io2-s2 kernel: Lustre: Skipped 3 previous similar messages
Apr 22 16:44:38 fir-io2-s2 kernel: Lustre: fir-OST0017: Client 9a46a636-d807-725a-1806-a4c05a6a1620 (at 10.8.18.24@o2ib6) reconnecting
Apr 22 16:44:38 fir-io2-s2 kernel: Lustre: Skipped 19 previous similar messages
Apr 22 16:44:42 fir-io2-s2 kernel: Lustre: fir-OST000f: Client 13af64a9-0ae4-d891-8650-3ffb19d9c8b0 (at 10.8.26.12@o2ib6) reconnecting
Apr 22 16:44:42 fir-io2-s2 kernel: Lustre: Skipped 29 previous similar messages
Apr 22 16:44:45 fir-io2-s2 kernel: Lustre: 84942:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555976678/real 0]  req@ffff90f760228c00 x1631536975368304/t0(0) o104->fir-OST0013@10.8.27.23@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555976685 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Apr 22 16:44:45 fir-io2-s2 kernel: Lustre: 84942:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 35 previous similar messages
Apr 22 16:44:47 fir-io2-s2 kernel: LustreError: 85222:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff90e29ca1dc50 x1631301490647712/t0(0) o4->1b540135-ca45-f45a-a248-eb5cb55b2913@10.8.29.4@o2ib6:229/0 lens 488/448 e 1 to 0 dl 1555976709 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 16:44:47 fir-io2-s2 kernel: Lustre: fir-OST0015: Bulk IO write error with 1b540135-ca45-f45a-a248-eb5cb55b2913 (at 10.8.29.4@o2ib6), client will retry: rc = -110
Apr 22 16:44:50 fir-io2-s2 kernel: Lustre: fir-OST0015: Client 13870419-853f-391c-b5d3-21baf53508af (at 10.8.14.6@o2ib6) reconnecting
Apr 22 16:44:50 fir-io2-s2 kernel: Lustre: Skipped 22 previous similar messages
Apr 22 16:45:01 fir-io2-s2 kernel: Lustre: fir-OST0013: haven't heard from client f3323a92-51ab-836f-42e3-c1fd4bff7b2d (at 10.9.102.45@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91126411c000, cur 1555976701 expire 1555976551 last 1555976474
Apr 22 16:45:01 fir-io2-s2 kernel: Lustre: Skipped 11 previous similar messages
Apr 22 16:45:06 fir-io2-s2 kernel: Lustre: fir-OST000f: Client 7bc968ec-0af0-1f9a-1f8e-b6540b9ff2ce (at 10.8.29.5@o2ib6) reconnecting
Apr 22 16:45:06 fir-io2-s2 kernel: Lustre: Skipped 62 previous similar messages
Apr 22 16:45:08 fir-io2-s2 kernel: LustreError: 85332:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff90e188bb0450 x1628578037680480/t0(0) o4->e0d09136-0a0d-4d3c-a8ef-7cfcd4b2ff05@10.8.2.24@o2ib6:245/0 lens 488/448 e 1 to 0 dl 1555976725 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 16:45:09 fir-io2-s2 kernel: LustreError: 137-5: fir-OST000c_UUID: not available for connect from 10.8.27.28@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 16:45:09 fir-io2-s2 kernel: LustreError: Skipped 1 previous similar message
Apr 22 16:45:09 fir-io2-s2 kernel: LustreError: 85008:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff90e29ca1d050 x1631354477947424/t0(0) o4->334d5bc3-122f-b515-8402-3988f1cde106@10.8.21.11@o2ib6:245/0 lens 488/448 e 1 to 0 dl 1555976725 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 16:45:09 fir-io2-s2 kernel: LustreError: 85008:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 3 previous similar messages
Apr 22 16:45:10 fir-io2-s2 kernel: LNetError: 81274:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 16:45:10 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f15a82cc00
Apr 22 16:45:10 fir-io2-s2 kernel: Lustre: fir-OST0015: Bulk IO write error with 1b540135-ca45-f45a-a248-eb5cb55b2913 (at 10.8.29.4@o2ib6), client will retry: rc = -110
Apr 22 16:45:10 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5aac76c00
Apr 22 16:45:10 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5aac76c00
Apr 22 16:45:10 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5aac76c00
Apr 22 16:45:10 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5aac76c00
Apr 22 16:45:10 fir-io2-s2 kernel: LNetError: 81268:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 16:45:10 fir-io2-s2 kernel: LNetError: 81268:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages
Apr 22 16:45:10 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d54d26fa00
Apr 22 16:45:10 fir-io2-s2 kernel: LustreError: 137-5: fir-OST000c_UUID: not available for connect from 10.8.26.17@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 16:45:10 fir-io2-s2 kernel: LustreError: Skipped 5 previous similar messages
Apr 22 16:45:12 fir-io2-s2 kernel: LustreError: 137-5: fir-OST0012_UUID: not available for connect from 10.8.8.1@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 16:45:12 fir-io2-s2 kernel: LustreError: Skipped 19 previous similar messages
Apr 22 16:45:12 fir-io2-s2 kernel: LNetError: 81272:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 16:45:12 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3c6c1bc00
Apr 22 16:45:12 fir-io2-s2 kernel: Lustre: fir-OST0015: Bulk IO read error with 7bd3af25-bee7-c5ba-c622-d03b941d9a42 (at 10.8.29.6@o2ib6), client will retry: rc -110
Apr 22 16:45:12 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f44e679000
Apr 22 16:45:12 fir-io2-s2 kernel: Lustre: fir-OST0011: Bulk IO write error with 253b4c5b-4ff4-3bf5-58fe-413737b1d5c2 (at 10.8.20.14@o2ib6), client will retry: rc = -110
Apr 22 16:45:12 fir-io2-s2 kernel: Lustre: Skipped 2 previous similar messages
Apr 22 16:45:13 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3cdd5ce00
Apr 22 16:45:14 fir-io2-s2 kernel: LustreError: 137-5: fir-OST0012_UUID: not available for connect from 10.8.21.31@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 16:45:14 fir-io2-s2 kernel: LustreError: Skipped 6 previous similar messages
Apr 22 16:45:15 fir-io2-s2 kernel: LNetError: 81274:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 16:45:15 fir-io2-s2 kernel: LNetError: 81274:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages
Apr 22 16:45:15 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dc22368a00
Apr 22 16:45:15 fir-io2-s2 kernel: Lustre: fir-OST0015: Bulk IO read error with 7bd3af25-bee7-c5ba-c622-d03b941d9a42 (at 10.8.29.6@o2ib6), client will retry: rc -110
Apr 22 16:45:17 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3cdd5ce00
Apr 22 16:45:17 fir-io2-s2 kernel: Lustre: fir-OST0015: Bulk IO write error with f2a4d35e-05ff-fe02-d6c7-6c183d27b8a1 (at 10.8.8.29@o2ib6), client will retry: rc = -110
Apr 22 16:45:17 fir-io2-s2 kernel: Lustre: 84888:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555976710/real 0]  req@ffff90f66b30e900 x1631536975640880/t0(0) o104->fir-OST0011@10.8.27.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555976717 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Apr 22 16:45:17 fir-io2-s2 kernel: Lustre: 84888:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 41 previous similar messages
Apr 22 16:45:18 fir-io2-s2 kernel: LustreError: 137-5: fir-OST000e_UUID: not available for connect from 10.8.27.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 16:45:18 fir-io2-s2 kernel: LustreError: Skipped 10 previous similar messages
Apr 22 16:45:28 fir-io2-s2 kernel: LustreError: 137-5: fir-OST000c_UUID: not available for connect from 10.8.7.19@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 16:45:28 fir-io2-s2 kernel: LustreError: Skipped 5 previous similar messages
Apr 22 16:45:38 fir-io2-s2 kernel: Lustre: fir-OST0011: Client 9c1d3e3f-d6ea-0750-3524-77592e0a32da (at 10.8.27.17@o2ib6) reconnecting
Apr 22 16:45:38 fir-io2-s2 kernel: Lustre: Skipped 127 previous similar messages
Apr 22 16:45:57 fir-io2-s2 kernel: LustreError: 137-5: fir-OST0012_UUID: not available for connect from 10.8.2.34@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 16:45:57 fir-io2-s2 kernel: LustreError: Skipped 52 previous similar messages
Apr 22 16:46:41 fir-io2-s2 kernel: LustreError: 137-5: fir-OST000c_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 16:46:41 fir-io2-s2 kernel: LustreError: Skipped 1 previous similar message
Apr 22 16:47:06 fir-io2-s2 kernel: Lustre: fir-OST000d: Client da84e749-aae7-e201-4af5-a27465c5b762 (at 10.8.0.65@o2ib6) reconnecting
Apr 22 16:47:06 fir-io2-s2 kernel: Lustre: Skipped 82 previous similar messages
Apr 22 16:47:28 fir-io2-s2 kernel: Lustre: fir-OST0011: Connection restored to da84e749-aae7-e201-4af5-a27465c5b762 (at 10.8.0.65@o2ib6)
Apr 22 16:47:28 fir-io2-s2 kernel: Lustre: Skipped 391 previous similar messages
Apr 22 16:48:01 fir-io2-s2 kernel: Lustre: 83729:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555976874/real 0]  req@ffff90f14447b300 x1631536977580784/t0(0) o104->fir-OST0011@10.8.23.2@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555976881 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Apr 22 16:48:01 fir-io2-s2 kernel: Lustre: 83729:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 32 previous similar messages
Apr 22 16:48:09 fir-io2-s2 kernel: LustreError: 137-5: fir-OST000c_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 16:48:09 fir-io2-s2 kernel: LustreError: Skipped 5 previous similar messages
Apr 22 16:48:29 fir-io2-s2 kernel: LustreError: 85107:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff90fd25569450 x1631542227769584/t0(0) o3->218c0296-144e-109b-969b-abd68eba00c8@10.8.26.35@o2ib6:448/0 lens 488/440 e 1 to 0 dl 1555976928 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 16:48:29 fir-io2-s2 kernel: LustreError: 85107:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 2 previous similar messages
Apr 22 16:48:30 fir-io2-s2 kernel: LNetError: 81274:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 16:48:30 fir-io2-s2 kernel: LNetError: 81274:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message
Apr 22 16:48:30 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df9664a200
Apr 22 16:48:30 fir-io2-s2 kernel: Lustre: fir-OST000f: Bulk IO read error with 218c0296-144e-109b-969b-abd68eba00c8 (at 10.8.26.35@o2ib6), client will retry: rc -110
Apr 22 16:48:30 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f635f42400
Apr 22 16:48:31 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fe2858f600
Apr 22 16:48:32 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7c1f1aa00
Apr 22 16:48:32 fir-io2-s2 kernel: Lustre: fir-OST000f: Bulk IO read error with 218c0296-144e-109b-969b-abd68eba00c8 (at 10.8.26.35@o2ib6), client will retry: rc -110
Apr 22 16:48:32 fir-io2-s2 kernel: Lustre: Skipped 2 previous similar messages
Apr 22 16:48:34 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e6e3a60a00
Apr 22 16:52:13 fir-io2-s2 kernel: Lustre: fir-OST000d: haven't heard from client a6c677eb-4ace-e7b4-4376-24acf7debcee (at 10.9.108.51@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91121c693400, cur 1555977133 expire 1555976983 last 1555976906
Apr 22 16:52:13 fir-io2-s2 kernel: Lustre: Skipped 5 previous similar messages
Apr 22 16:53:05 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.8.11.15@o2ib6  ns: filter-fir-OST0017_UUID lock: ffff90f288811b00/0x4ec9aa14ebd36ce0 lrc: 3/0,0 mode: PW/PW res: [0x940000400:0x3ea84d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400020020 nid: 10.8.11.15@o2ib6 remote: 0x7732101709dabd2f expref: 6 pid: 83787 timeout: 23357 lvb_type: 0
Apr 22 16:53:05 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages
Apr 22 16:58:27 fir-io2-s2 kernel: Lustre: fir-OST0013: Client da84e749-aae7-e201-4af5-a27465c5b762 (at 10.8.0.65@o2ib6) reconnecting
Apr 22 16:58:27 fir-io2-s2 kernel: Lustre: Skipped 18 previous similar messages
Apr 22 16:58:27 fir-io2-s2 kernel: Lustre: fir-OST0013: Connection restored to da84e749-aae7-e201-4af5-a27465c5b762 (at 10.8.0.65@o2ib6)
Apr 22 16:58:27 fir-io2-s2 kernel: Lustre: Skipped 31 previous similar messages
Apr 22 16:59:46 fir-io2-s2 kernel: LustreError: 137-5: fir-OST0016_UUID: not available for connect from 10.8.0.65@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 16:59:46 fir-io2-s2 kernel: LustreError: Skipped 3 previous similar messages
Apr 22 17:00:06 fir-io2-s2 kernel: Lustre: 84985:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555977599/real 0]  req@ffff9112859c0000 x1631536985569248/t0(0) o104->fir-OST0011@10.8.27.9@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555977606 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Apr 22 17:00:06 fir-io2-s2 kernel: Lustre: 84985:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 66 previous similar messages
Apr 22 17:00:07 fir-io2-s2 kernel: Lustre: fir-OST0017: haven't heard from client 24a34926-dd8c-21dc-b1ff-31d591ccefa0 (at 10.9.102.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9112577a7000, cur 1555977607 expire 1555977457 last 1555977380
Apr 22 17:00:07 fir-io2-s2 kernel: Lustre: Skipped 18 previous similar messages
Apr 22 17:05:36 fir-io2-s2 kernel: Lustre: fir-OST0011: Client 9d21700a-8b98-c7af-b3fd-bc1cec65d723 (at 10.8.8.37@o2ib6) reconnecting
Apr 22 17:05:36 fir-io2-s2 kernel: Lustre: Skipped 646 previous similar messages
Apr 22 17:05:48 fir-io2-s2 kernel: Lustre: 85017:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555977941/real 0]  req@ffff90fb245fcb00 x1631536991068080/t0(0) o104->fir-OST0015@10.8.28.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555977948 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Apr 22 17:05:48 fir-io2-s2 kernel: Lustre: 85017:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Apr 22 17:06:16 fir-io2-s2 kernel: LustreError: 137-5: fir-OST0012_UUID: not available for connect from 10.8.17.15@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 17:06:16 fir-io2-s2 kernel: LustreError: Skipped 39 previous similar messages
Apr 22 17:06:17 fir-io2-s2 kernel: LustreError: 85447:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff90f0b3952450 x1631534848236816/t0(0) o4->13458280-a046-3a7f-2bec-0301aba013a1@10.8.28.12@o2ib6:25/0 lens 488/448 e 0 to 0 dl 1555978015 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:06:17 fir-io2-s2 kernel: LustreError: 85447:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 4 previous similar messages
Apr 22 17:06:30 fir-io2-s2 kernel: LNetError: 81269:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 17:06:30 fir-io2-s2 kernel: LNetError: 81269:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages
Apr 22 17:06:30 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90edddf02000
Apr 22 17:06:30 fir-io2-s2 kernel: Lustre: fir-OST000d: Bulk IO write error with 13458280-a046-3a7f-2bec-0301aba013a1 (at 10.8.28.12@o2ib6), client will retry: rc = -110
Apr 22 17:07:42 fir-io2-s2 kernel: LustreError: 85235:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff90e2bb7d9c50 x1631300298728000/t0(0) o3->53503b0c-0e39-21a5-cb74-36c9daf2fc04@10.8.13.20@o2ib6:86/0 lens 488/440 e 0 to 0 dl 1555978076 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:07:47 fir-io2-s2 kernel: LustreError: 85139:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff90f0b3955850 x1631293649497824/t0(0) o3->1f782456-cd04-17fc-9c8a-d25006994669@10.8.1.4@o2ib6:119/0 lens 488/440 e 0 to 0 dl 1555978109 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:07:47 fir-io2-s2 kernel: LustreError: 85139:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 2 previous similar messages
Apr 22 17:07:50 fir-io2-s2 kernel: LustreError: 84995:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff910275b5cc50 x1631380129586112/t0(0) o3->755dec3b-dc25-b089-696c-73e1e6537c3e@10.8.10.33@o2ib6:118/0 lens 488/440 e 0 to 0 dl 1555978108 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:07:50 fir-io2-s2 kernel: LustreError: 84995:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 4 previous similar messages
Apr 22 17:07:51 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.8.0.67@o2ib6  ns: filter-fir-OST0011_UUID lock: ffff90d90590e540/0x4ec9aa14ee911380 lrc: 3/0,0 mode: PR/PR res: [0x680000400:0x190555:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400010020 nid: 10.8.0.67@o2ib6 remote: 0x997259fc399f375d expref: 459 pid: 83760 timeout: 24243 lvb_type: 1
Apr 22 17:07:55 fir-io2-s2 kernel: LustreError: 85330:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff90f0b3954450 x1631545734048880/t0(0) o3->8776df39-692a-3df9-0874-72e441440742@10.8.18.22@o2ib6:116/0 lens 488/440 e 1 to 0 dl 1555978106 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:07:55 fir-io2-s2 kernel: LustreError: 85330:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 16 previous similar messages
Apr 22 17:07:56 fir-io2-s2 kernel: LNetError: 81278:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 17:07:56 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fced4a2c00
Apr 22 17:07:56 fir-io2-s2 kernel: Lustre: fir-OST0011: Bulk IO read error with 4a5b9079-3e8c-70ae-a88f-6098c9496628 (at 10.8.28.4@o2ib6), client will retry: rc -110
Apr 22 17:07:56 fir-io2-s2 kernel: Lustre: Skipped 1 previous similar message
Apr 22 17:07:58 fir-io2-s2 kernel: LNetError: 81274:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 17:07:58 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f90ada4000
Apr 22 17:07:58 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f90ada4000
Apr 22 17:07:58 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f90ada4000
Apr 22 17:07:58 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f90ada4000
Apr 22 17:07:58 fir-io2-s2 kernel: LNetError: 81274:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages
Apr 22 17:07:58 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f90ada4000
Apr 22 17:07:59 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f90ada4000
Apr 22 17:07:59 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f90ada4000
Apr 22 17:07:59 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f90ada4000
Apr 22 17:07:59 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f90ada4000
Apr 22 17:07:59 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f90ada4000
Apr 22 17:07:59 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f90ada4000
Apr 22 17:07:59 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f90ada4000
Apr 22 17:07:59 fir-io2-s2 kernel: Lustre: fir-OST000f: Bulk IO write error with 61a62562-ad73-dc2c-c4dd-497b185dd54e (at 10.8.16.5@o2ib6), client will retry: rc = -110
Apr 22 17:07:59 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7aa962800
Apr 22 17:07:59 fir-io2-s2 kernel: Lustre: fir-OST0017: Bulk IO read error with 755dec3b-dc25-b089-696c-73e1e6537c3e (at 10.8.10.33@o2ib6), client will retry: rc -110
Apr 22 17:08:03 fir-io2-s2 kernel: LustreError: 85002:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff90e2b59c9450 x1631377036965280/t0(0) o3->193df16c-8b67-30a8-f3e5-c6f520fad888@10.8.11.20@o2ib6:123/0 lens 488/440 e 1 to 0 dl 1555978113 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:08:03 fir-io2-s2 kernel: LustreError: 85002:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 5 previous similar messages
Apr 22 17:08:04 fir-io2-s2 kernel: LNetError: 81277:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 17:08:04 fir-io2-s2 kernel: LNetError: 81277:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 8 previous similar messages
Apr 22 17:08:04 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90defb617000
Apr 22 17:08:04 fir-io2-s2 kernel: Lustre: fir-OST000d: Bulk IO read error with 53503b0c-0e39-21a5-cb74-36c9daf2fc04 (at 10.8.13.20@o2ib6), client will retry: rc -110
Apr 22 17:08:05 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e29e74c600
Apr 22 17:08:05 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e70eda4800
Apr 22 17:08:06 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec2bc17400
Apr 22 17:08:06 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7e77be600
Apr 22 17:08:06 fir-io2-s2 kernel: Lustre: fir-OST0017: Bulk IO read error with b918e791-1350-4c1c-cede-16b78e9f3d1a (at 10.8.11.19@o2ib6), client will retry: rc -110
Apr 22 17:08:06 fir-io2-s2 kernel: Lustre: Skipped 3 previous similar messages
Apr 22 17:08:08 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927094600
Apr 22 17:08:10 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e4a47c8600
Apr 22 17:08:11 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f12c226800
Apr 22 17:08:11 fir-io2-s2 kernel: Lustre: fir-OST0011: Bulk IO read error with 4a5b9079-3e8c-70ae-a88f-6098c9496628 (at 10.8.28.4@o2ib6), client will retry: rc -110
Apr 22 17:08:11 fir-io2-s2 kernel: Lustre: Skipped 2 previous similar messages
Apr 22 17:08:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f684b0a800
Apr 22 17:08:13 fir-io2-s2 kernel: LNetError: 81268:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 17:08:13 fir-io2-s2 kernel: LNetError: 81268:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 8 previous similar messages
Apr 22 17:08:13 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da8f4c3a00
Apr 22 17:08:14 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927095800
Apr 22 17:08:14 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f684b0c200
Apr 22 17:08:15 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9100f78db400
Apr 22 17:08:19 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e5c8c55000
Apr 22 17:08:19 fir-io2-s2 kernel: Lustre: fir-OST0017: Bulk IO read error with 755dec3b-dc25-b089-696c-73e1e6537c3e (at 10.8.10.33@o2ib6), client will retry: rc -110
Apr 22 17:08:19 fir-io2-s2 kernel: Lustre: Skipped 5 previous similar messages
Apr 22 17:08:20 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d646fd5800
Apr 22 17:08:22 fir-io2-s2 kernel: LustreError: 85234:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff90f2bb5d4450 x1631343916533648/t0(0) o3->7bd3af25-bee7-c5ba-c622-d03b941d9a42@10.8.29.6@o2ib6:119/0 lens 488/440 e 1 to 0 dl 1555978109 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:08:22 fir-io2-s2 kernel: LustreError: 85234:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 11 previous similar messages
Apr 22 17:08:23 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ee00479000
Apr 22 17:08:23 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dd52c54c00
Apr 22 17:08:25 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e9db756400
Apr 22 17:08:26 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1b0d43e00
Apr 22 17:08:26 fir-io2-s2 kernel: Lustre: fir-OST0015: Bulk IO write error with f3bba4e8-9568-4001-6257-88537741a8c9 (at 10.8.29.3@o2ib6), client will retry: rc = -110
Apr 22 17:08:26 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f09588be00
Apr 22 17:08:27 fir-io2-s2 kernel: Lustre: fir-OST0011: Connection restored to 7bc968ec-0af0-1f9a-1f8e-b6540b9ff2ce (at 10.8.29.5@o2ib6)
Apr 22 17:08:27 fir-io2-s2 kernel: Lustre: Skipped 917 previous similar messages
Apr 22 17:08:28 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db59f8a400
Apr 22 17:08:31 fir-io2-s2 kernel: LNetError: 81274:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 17:08:31 fir-io2-s2 kernel: LNetError: 81274:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 11 previous similar messages
Apr 22 17:08:31 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ee33785a00
Apr 22 17:08:32 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f536d39c00
Apr 22 17:08:33 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa8a913e00
Apr 22 17:08:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e188aafa00
Apr 22 17:08:34 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90daebcb8400
Apr 22 17:08:34 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de8bb60200
Apr 22 17:08:35 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dd12691400
Apr 22 17:08:37 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff910080bd1600
Apr 22 17:08:37 fir-io2-s2 kernel: Lustre: fir-OST000f: Bulk IO read error with d36980b7-2b04-f724-0e6b-cf989e4d7da2 (at 10.8.1.34@o2ib6), client will retry: rc -110
Apr 22 17:08:37 fir-io2-s2 kernel: Lustre: Skipped 14 previous similar messages
Apr 22 17:08:39 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f536d3ba00
Apr 22 17:08:39 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff910080bd2a00
Apr 22 17:08:40 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da5e883e00
Apr 22 17:08:40 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e14a808400
Apr 22 17:08:44 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ee75a46e00
Apr 22 17:08:47 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3e4a4ec00
Apr 22 17:08:47 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db2212a000
Apr 22 17:08:49 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb7e468a00
Apr 22 17:08:49 fir-io2-s2 kernel: Lustre: fir-OST0013: Bulk IO write error with f2a4d35e-05ff-fe02-d6c7-6c183d27b8a1 (at 10.8.8.29@o2ib6), client will retry: rc = -110
Apr 22 17:08:51 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7f5c79200
Apr 22 17:08:52 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e09ee02800
Apr 22 17:08:54 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fc478dcc00
Apr 22 17:08:55 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d6168ec800
Apr 22 17:08:55 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec61a02e00
Apr 22 17:08:59 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e94cc4a000
Apr 22 17:08:59 fir-io2-s2 kernel: LustreError: 85295:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff90ebe154c450 x1631293631807152/t0(0) o4->301bd1d5-f294-00c7-57bb-0517ce6cb157@10.8.15.8@o2ib6:171/0 lens 488/448 e 2 to 0 dl 1555978161 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:08:59 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec61a02e00
Apr 22 17:08:59 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec61a02e00
Apr 22 17:08:59 fir-io2-s2 kernel: Lustre: fir-OST0011: Bulk IO write error with 301bd1d5-f294-00c7-57bb-0517ce6cb157 (at 10.8.15.8@o2ib6), client will retry: rc = -110
Apr 22 17:08:59 fir-io2-s2 kernel: LustreError: 85295:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 1 previous similar message
Apr 22 17:08:59 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ed73e94200
Apr 22 17:09:04 fir-io2-s2 kernel: LNetError: 81275:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 17:09:04 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:04 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:04 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:04 fir-io2-s2 kernel: LNetError: 81275:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 27 previous similar messages
Apr 22 17:09:04 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:08 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:08 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:08 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:08 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:08 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:08 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:08 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:08 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:09 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e018183a00
Apr 22 17:09:09 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:09 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:10 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:10 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e927709e00
Apr 22 17:09:10 fir-io2-s2 kernel: LustreError: 85009:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff90f10dadcc50 x1631542370227568/t0(0) o4->69e867f7-2c34-9281-0411-6ff880d43ef5@10.8.28.11@o2ib6:182/0 lens 488/448 e 2 to 0 dl 1555978172 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:09:10 fir-io2-s2 kernel: LustreError: 85009:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 1 previous similar message
Apr 22 17:09:10 fir-io2-s2 kernel: Lustre: fir-OST0011: Bulk IO write error with 69e867f7-2c34-9281-0411-6ff880d43ef5 (at 10.8.28.11@o2ib6), client will retry: rc = -110
Apr 22 17:09:10 fir-io2-s2 kernel: Lustre: Skipped 2 previous similar messages
Apr 22 17:09:12 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e018183a00
Apr 22 17:09:12 fir-io2-s2 kernel: LustreError: 85376:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk READ  req@ffff90e1d1d32c50 x1631530955540112/t0(0) o3->220c5a2f-2274-34f0-c226-fa1f26d20160@10.8.1.29@o2ib6:185/0 lens 488/440 e 1 to 0 dl 1555978175 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:09:12 fir-io2-s2 kernel: Lustre: fir-OST0011: Bulk IO read error with 220c5a2f-2274-34f0-c226-fa1f26d20160 (at 10.8.1.29@o2ib6), client will retry: rc -110
Apr 22 17:09:12 fir-io2-s2 kernel: Lustre: Skipped 11 previous similar messages
Apr 22 17:09:12 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb263f2600
Apr 22 17:09:12 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb263f2600
Apr 22 17:09:15 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e17ff30e00
Apr 22 17:09:28 fir-io2-s2 kernel: LustreError: 85326:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff90ebe154e050 x1631530955551024/t0(0) o3->220c5a2f-2274-34f0-c226-fa1f26d20160@10.8.1.29@o2ib6:253/0 lens 488/440 e 0 to 0 dl 1555978243 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:09:28 fir-io2-s2 kernel: LustreError: 85326:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 3 previous similar messages
Apr 22 17:09:46 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d636afd600
Apr 22 17:09:46 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d636afd600
Apr 22 17:09:46 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d636afd600
Apr 22 17:09:46 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d636afd600
Apr 22 17:09:46 fir-io2-s2 kernel: Lustre: fir-OST0011: Bulk IO write error with be42b497-ab1b-8d58-3101-014aad577cfc (at 10.8.27.35@o2ib6), client will retry: rc = -110
Apr 22 17:09:47 fir-io2-s2 kernel: Lustre: Skipped 1 previous similar message
Apr 22 17:09:54 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d96f968200
Apr 22 17:09:54 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d96f968200
Apr 22 17:09:54 fir-io2-s2 kernel: LustreError: 85052:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff90e2b59c9050 x1631318897305968/t0(0) o4->cf702945-c76f-86a0-028f-9ef2c86b3cdd@10.8.10.15@o2ib6:223/0 lens 488/448 e 1 to 0 dl 1555978213 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:09:54 fir-io2-s2 kernel: LustreError: 85052:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 1 previous similar message
Apr 22 17:09:59 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f5904bd800
Apr 22 17:09:59 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d4e91f3600
Apr 22 17:09:59 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d4e91f3600
Apr 22 17:09:59 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d4e91f3600
Apr 22 17:09:59 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d4e91f3600
Apr 22 17:09:59 fir-io2-s2 kernel: LustreError: 85274:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff90e29faac850 x1631318897306144/t0(0) o4->cf702945-c76f-86a0-028f-9ef2c86b3cdd@10.8.10.15@o2ib6:223/0 lens 488/448 e 1 to 0 dl 1555978213 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:10:22 fir-io2-s2 kernel: LNetError: 81273:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 17:10:22 fir-io2-s2 kernel: LNetError: 81273:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 30 previous similar messages
Apr 22 17:10:29 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507830600
Apr 22 17:10:29 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507830600
Apr 22 17:10:29 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507830600
Apr 22 17:10:29 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507830600
Apr 22 17:10:29 fir-io2-s2 kernel: LustreError: 85247:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff90ebe154a850 x1631541844517744/t0(0) o4->aba5d4eb-e07c-9b0f-6ab5-7f97caf38a26@10.8.16.4@o2ib6:256/0 lens 488/448 e 1 to 0 dl 1555978246 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:10:29 fir-io2-s2 kernel: LustreError: 85247:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 1 previous similar message
Apr 22 17:10:29 fir-io2-s2 kernel: Lustre: fir-OST0017: Bulk IO write error with aba5d4eb-e07c-9b0f-6ab5-7f97caf38a26 (at 10.8.16.4@o2ib6), client will retry: rc = -110
Apr 22 17:10:29 fir-io2-s2 kernel: Lustre: Skipped 3 previous similar messages
Apr 22 17:10:32 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e96f050000
Apr 22 17:10:32 fir-io2-s2 kernel: Lustre: fir-OST0011: Bulk IO read error with 220c5a2f-2274-34f0-c226-fa1f26d20160 (at 10.8.1.29@o2ib6), client will retry: rc -110
Apr 22 17:10:32 fir-io2-s2 kernel: Lustre: Skipped 1 previous similar message
Apr 22 17:10:56 fir-io2-s2 kernel: LustreError: 84916:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff90fd19967850 x1631532303155456/t0(0) o4->d3013375-2e90-b76e-c4d8-76867f2b4a32@10.8.2.20@o2ib6:297/0 lens 488/448 e 1 to 0 dl 1555978287 ref 1 fl Interpret:/2/0 rc 0/0
Apr 22 17:11:19 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Apr 22 17:11:19 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.202@o2ib7 (0): c: 0, oc: 0, rc: 8
Apr 22 17:11:26 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds
Apr 22 17:11:26 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 1 previous similar message
Apr 22 17:11:26 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.202@o2ib7 (0): c: 0, oc: 1, rc: 8
Apr 22 17:11:26 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 1 previous similar message
Apr 22 17:11:30 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Apr 22 17:11:30 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.204@o2ib7 (0): c: 0, oc: 0, rc: 8
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 84945:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.7.35@o2ib6) failed to reply to blocking AST (req@ffff9112b4db0900 x1631536993878544 status 0 rc -110), evict it ns: filter-fir-OST000d_UUID lock: ffff90efc9953180/0x4ec9aa14eb9cbe99 lrc: 4/0,0 mode: PR/PR res: [0x300000402:0x92066b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x60000400000020 nid: 10.8.7.35@o2ib6 remote: 0xf7bdc79e22c2cf06 expref: 1503 pid: 84818 timeout: 24457 lvb_type: 1
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f0bebe7e00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f43de35400
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f43de35400
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f43de35400
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f43de35400
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7fdc93800
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7fdc93800
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7fdc93800
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7fdc93800
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 85185:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff90ebe154e450 x1631541884346160/t0(0) o4->61a62562-ad73-dc2c-c4dd-497b185dd54e@10.8.16.5@o2ib6:327/0 lens 488/448 e 2 to 0 dl 1555978317 ref 1 fl Interpret:/2/0 rc 0/0
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7fdc93800
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7fdc93800
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7fdc93800
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7fdc93800
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7fdc93800
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7fdc93800
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7fdc93800
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7fdc93800
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 84945:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: 138-a: fir-OST000d: A client on nid 10.8.7.35@o2ib6 was evicted due to a lock blocking callback time out: rc -110
Apr 22 17:11:32 fir-io2-s2 kernel: LustreError: Skipped 3 previous similar messages
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 85158:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 100+0s  req@ffff90e2bb7d9c50 x1631541884349200/t0(0) o4->61a62562-ad73-dc2c-c4dd-497b185dd54e@10.8.16.5@o2ib6:327/0 lens 488/448 e 2 to 0 dl 1555978317 ref 1 fl Interpret:/2/0 rc 0/0
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 85158:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 1 previous similar message
Apr 22 17:11:37 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Apr 22 17:11:37 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.202@o2ib7 (0): c: 0, oc: 3, rc: 6
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90e726026a00
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90e726026a00
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90e726026a00
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90e726026a00
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90e726026a00
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90e726026a00
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90f4602bf000
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90f468d9f200
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90e726026a00
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90e726026a00
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90e726026a00
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90e726026a00
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90e726026a00
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90eb379fa400
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90f4602bd400
Apr 22 17:11:37 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90f76202b400
Apr 22 17:11:42 fir-io2-s2 kernel: Lustre: fir-OST0011: haven't heard from client 2e220de5-7b5b-3874-45ea-64c959a50d0b (at 10.8.0.67@o2ib6) in 180 seconds. I think it's dead, and I am evicting it. exp ffff90e189d9f000, cur 1555978302 expire 1555978152 last 1555978122
Apr 22 17:11:42 fir-io2-s2 kernel: Lustre: Skipped 53 previous similar messages
Apr 22 17:11:43 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds
Apr 22 17:11:43 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 1 previous similar message
Apr 22 17:11:43 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.202@o2ib7 (1): c: 0, oc: 3, rc: 7
Apr 22 17:11:43 fir-io2-s2 kernel: LNetError: 81266:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 1 previous similar message
Apr 22 17:11:43 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90f762028200
Apr 22 17:11:47 fir-io2-s2 kernel: LustreError: 85009:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 100+0s  req@ffff90eff59b2c50 x1631320591904256/t0(0) o4->ccc8b821-13fa-5ca1-49da-be1a6cb0e16f@10.8.30.27@o2ib6:337/0 lens 488/448 e 1 to 0 dl 1555978327 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:11:50 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90fae2108a00
Apr 22 17:11:50 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90ec958f5400
Apr 22 17:11:50 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90fae210b400
Apr 22 17:11:50 fir-io2-s2 kernel: LustreError: 81266:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff90f53e720000
Apr 22 17:11:54 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 211s: evicting client at 10.8.11.26@o2ib6  ns: filter-fir-OST000f_UUID lock: ffff911265572f40/0x4ec9aa14bf7fe7fc lrc: 3/0,0 mode: PW/PW res: [0xac0000402:0x560d4b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000400010020 nid: 10.8.11.26@o2ib6 remote: 0xa234b98ee14de6df expref: 6 pid: 82562 timeout: 24486 lvb_type: 0
Apr 22 17:12:27 fir-io2-s2 kernel: LNet: Service thread pid 83769 was inactive for 200.01s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Apr 22 17:12:27 fir-io2-s2 kernel: LNet: Skipped 1 previous similar message
Apr 22 17:12:27 fir-io2-s2 kernel: Pid: 83769, comm: ll_ost01_036 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 17:12:27 fir-io2-s2 kernel: Call Trace:
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffc1232640>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc]
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffc11effe5>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc]
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffc121169b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc]
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffc181310b>] ofd_intent_policy+0x69b/0x920 [ofd]
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffc11f0d26>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc]
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffc12196d7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc]
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffc12a00b2>] tgt_enqueue+0x62/0x210 [ptlrpc]
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 17:12:27 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 17:12:27 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555978347.83769
Apr 22 17:12:28 fir-io2-s2 kernel: LNet: Service thread pid 83747 was inactive for 201.08s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Apr 22 17:12:28 fir-io2-s2 kernel: Pid: 83747, comm: ll_ost01_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 17:12:28 fir-io2-s2 kernel: Call Trace:
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc1232640>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc11effe5>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc121169b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc181310b>] ofd_intent_policy+0x69b/0x920 [ofd]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc11f0d26>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc12196d7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc12a00b2>] tgt_enqueue+0x62/0x210 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 17:12:28 fir-io2-s2 kernel: Pid: 83781, comm: ll_ost01_041 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 17:12:28 fir-io2-s2 kernel: Call Trace:
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc1232640>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc11effe5>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc121169b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc181310b>] ofd_intent_policy+0x69b/0x920 [ofd]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc11f0d26>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc12196d7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc12a00b2>] tgt_enqueue+0x62/0x210 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 17:12:28 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 17:12:38 fir-io2-s2 kernel: LustreError: 85169:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk READ  req@ffff90ef4cd1b450 x1631300298755600/t0(0) o3->53503b0c-0e39-21a5-cb74-36c9daf2fc04@10.8.13.20@o2ib6:457/0 lens 488/440 e 0 to 0 dl 1555978447 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:12:38 fir-io2-s2 kernel: Lustre: fir-OST000f: Bulk IO write error with 69e867f7-2c34-9281-0411-6ff880d43ef5 (at 10.8.28.11@o2ib6), client will retry: rc = -110
Apr 22 17:12:38 fir-io2-s2 kernel: Lustre: Skipped 3 previous similar messages
Apr 22 17:12:38 fir-io2-s2 kernel: LustreError: 85169:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 11 previous similar messages
Apr 22 17:12:47 fir-io2-s2 kernel: LNetError: 81272:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 17:12:47 fir-io2-s2 kernel: LNetError: 81272:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 46 previous similar messages
Apr 22 17:12:47 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:12:53 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:12:53 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:12:53 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:12:53 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:12:53 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fedd133c00
Apr 22 17:12:53 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:12:53 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:12:53 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:12:53 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:12:54 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da74f1d200
Apr 22 17:13:01 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa9b4ee000
Apr 22 17:13:01 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa9b4ee000
Apr 22 17:13:01 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa9b4ee000
Apr 22 17:13:08 fir-io2-s2 kernel: LustreError: 85053:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff90f06b21d850 x1631377037430112/t0(0) o3->193df16c-8b67-30a8-f3e5-c6f520fad888@10.8.11.20@o2ib6:614/0 lens 488/440 e 0 to 0 dl 1555978604 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:13:08 fir-io2-s2 kernel: LustreError: 85053:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 2 previous similar messages
Apr 22 17:13:11 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f01f103400
Apr 22 17:13:14 fir-io2-s2 kernel: LNet: Service thread pid 83796 was inactive for 200.68s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Apr 22 17:13:14 fir-io2-s2 kernel: LNet: Skipped 1 previous similar message
Apr 22 17:13:14 fir-io2-s2 kernel: Pid: 83796, comm: ll_ost01_049 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 17:13:14 fir-io2-s2 kernel: Call Trace:
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffc1232640>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc]
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffc11effe5>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc]
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffc121169b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc]
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffc181310b>] ofd_intent_policy+0x69b/0x920 [ofd]
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffc11f0d26>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc]
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffc12196d7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc]
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffc12a00b2>] tgt_enqueue+0x62/0x210 [ptlrpc]
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 17:13:14 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 17:13:14 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555978394.83796
Apr 22 17:13:29 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f0e627b400
Apr 22 17:13:29 fir-io2-s2 kernel: Lustre: fir-OST0017: Bulk IO read error with 2c0bfc93-71cb-f565-f1fb-8f804a23ec4c (at 10.8.1.26@o2ib6), client will retry: rc -110
Apr 22 17:13:29 fir-io2-s2 kernel: Lustre: Skipped 10 previous similar messages
Apr 22 17:13:32 fir-io2-s2 kernel: LNet: Service thread pid 83796 completed after 219.36s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Apr 22 17:13:35 fir-io2-s2 kernel: LNet: Service thread pid 83769 completed after 268.13s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Apr 22 17:13:38 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 145s: evicting client at 10.8.14.9@o2ib6  ns: filter-fir-OST0013_UUID lock: ffff90e5be39c5c0/0x4ec9aa14e914cf87 lrc: 3/0,0 mode: PR/PR res: [0x600000400:0x57430d:0x0].0x0 rrc: 12 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400010020 nid: 10.8.14.9@o2ib6 remote: 0x123113cd124b9d4f expref: 883 pid: 83759 timeout: 24544 lvb_type: 1
Apr 22 17:13:41 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e9340e5c00
Apr 22 17:13:41 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df1043ae00
Apr 22 17:13:41 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df1043ae00
Apr 22 17:13:41 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df1043ae00
Apr 22 17:13:41 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df1043ae00
Apr 22 17:13:41 fir-io2-s2 kernel: LNet: Service thread pid 84840 was inactive for 260.21s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Apr 22 17:13:41 fir-io2-s2 kernel: Pid: 84840, comm: ll_ost02_046 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 17:13:41 fir-io2-s2 kernel: Call Trace:
Apr 22 17:13:41 fir-io2-s2 kernel: [<ffffffffc1232640>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc]
Apr 22 17:13:41 fir-io2-s2 kernel: [<ffffffffc11effe5>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc]
Apr 22 17:13:41 fir-io2-s2 kernel: [<ffffffffc11f0710>] ldlm_handle_conflict_lock+0x70/0x320 [ptlrpc]
Apr 22 17:13:41 fir-io2-s2 kernel: [<ffffffffc11f0ca3>] ldlm_lock_enqueue+0x2e3/0xa60 [ptlrpc]
Apr 22 17:13:41 fir-io2-s2 kernel: [<ffffffffc12196d7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc]
Apr 22 17:13:41 fir-io2-s2 kernel: [<ffffffffc12a00b2>] tgt_enqueue+0x62/0x210 [ptlrpc]
Apr 22 17:13:41 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 17:13:41 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 17:13:41 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 17:13:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 17:13:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 17:13:41 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 17:13:41 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555978421.84840
Apr 22 17:13:42 fir-io2-s2 kernel: LNet: Service thread pid 83747 completed after 275.05s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Apr 22 17:13:42 fir-io2-s2 kernel: LNet: Skipped 1 previous similar message
Apr 22 17:13:42 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.8.28.11@o2ib6  ns: filter-fir-OST0013_UUID lock: ffff90ff20ed7980/0x4ec9aa14d8a383a3 lrc: 4/0,0 mode: PR/PR res: [0x600000400:0x57430d:0x0].0x0 rrc: 11 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400010020 nid: 10.8.28.11@o2ib6 remote: 0xfc47ba201b7aff2c expref: 687 pid: 85027 timeout: 24594 lvb_type: 1
Apr 22 17:13:42 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 11 previous similar messages
Apr 22 17:13:51 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e885e0d000
Apr 22 17:13:53 fir-io2-s2 kernel: LNet: Service thread pid 84840 completed after 271.58s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Apr 22 17:13:56 fir-io2-s2 kernel: LNet: Service thread pid 84753 was inactive for 200.46s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 17:13:56 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555978436.84753
Apr 22 17:13:57 fir-io2-s2 kernel: LNet: Service thread pid 83786 was inactive for 200.84s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 17:13:57 fir-io2-s2 kernel: LustreError: 82362:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.1.18@o2ib6 arrived at 1555978437 with bad export cookie 5677255811366071855
Apr 22 17:13:57 fir-io2-s2 kernel: LustreError: 82362:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 3 previous similar messages
Apr 22 17:13:58 fir-io2-s2 kernel: LNet: Service thread pid 84799 was inactive for 200.56s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 17:13:58 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555978438.84799
Apr 22 17:13:58 fir-io2-s2 kernel: LustreError: 24834:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.1.22@o2ib6 arrived at 1555978438 with bad export cookie 5677255811366090909
Apr 22 17:13:58 fir-io2-s2 kernel: LustreError: 24834:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 1 previous similar message
Apr 22 17:14:00 fir-io2-s2 kernel: LustreError: 109378:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.1.22@o2ib6 arrived at 1555978440 with bad export cookie 5677255811366081039
Apr 22 17:14:00 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e385735000
Apr 22 17:14:00 fir-io2-s2 kernel: LustreError: 85330:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk READ  req@ffff90f1196f3850 x1631558794663056/t0(0) o3->2c0bfc93-71cb-f565-f1fb-8f804a23ec4c@10.8.1.26@o2ib6:601/0 lens 488/440 e 0 to 0 dl 1555978591 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:14:00 fir-io2-s2 kernel: LustreError: 85330:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 1 previous similar message
Apr 22 17:14:02 fir-io2-s2 kernel: LustreError: 100044:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.29.7@o2ib6 arrived at 1555978442 with bad export cookie 5677255811366091854
Apr 22 17:14:02 fir-io2-s2 kernel: LustreError: 100044:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 11 previous similar messages
Apr 22 17:14:08 fir-io2-s2 kernel: Lustre: fir-OST0011: Client 7bc968ec-0af0-1f9a-1f8e-b6540b9ff2ce (at 10.8.29.5@o2ib6) reconnecting
Apr 22 17:14:08 fir-io2-s2 kernel: Lustre: Skipped 1145 previous similar messages
Apr 22 17:14:08 fir-io2-s2 kernel: LustreError: 84995:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk READ after 100+0s  req@ffff9100e50a2050 x1631545916134512/t0(0) o3->d8d6f8e7-a2cd-08f2-c263-fa8b0dbeef3c@10.8.8.2@o2ib6:513/0 lens 488/440 e 0 to 0 dl 1555978503 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:14:08 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e32dd99600
Apr 22 17:14:08 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dff250f000
Apr 22 17:14:10 fir-io2-s2 kernel: LNet: Service thread pid 84779 was inactive for 200.46s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 17:14:10 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555978450.84779
Apr 22 17:14:11 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555978451.84777
Apr 22 17:14:11 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f059c08400
Apr 22 17:14:12 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f561f11e00
Apr 22 17:14:13 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e8bf8ffc00
Apr 22 17:14:13 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555978453.84896
Apr 22 17:14:14 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fef5c56a00
Apr 22 17:14:15 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fef5c56a00
Apr 22 17:14:15 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fef5c56a00
Apr 22 17:14:17 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e8e5099600
Apr 22 17:14:18 fir-io2-s2 kernel: LNet: Service thread pid 84896 completed after 264.53s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Apr 22 17:14:19 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1440ae800
Apr 22 17:14:20 fir-io2-s2 kernel: Lustre: 83769:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555978453/real 0]  req@ffff90f1695b3900 x1631536996880992/t0(0) o104->fir-OST000f@10.8.22.12@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555978460 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Apr 22 17:14:20 fir-io2-s2 kernel: Lustre: 83769:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 241 previous similar messages
Apr 22 17:14:20 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff2b413400
Apr 22 17:14:22 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e8bf8ffc00
Apr 22 17:14:23 fir-io2-s2 kernel: LustreError: 108922:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.7.23@o2ib6 arrived at 1555978463 with bad export cookie 5677255811366085911
Apr 22 17:14:23 fir-io2-s2 kernel: LustreError: 108922:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 7 previous similar messages
Apr 22 17:14:24 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f664cc7800
Apr 22 17:14:25 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db975b3e00
Apr 22 17:14:27 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e51750f600
Apr 22 17:14:30 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7f4a36000
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e65819ea00
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e65819ea00
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e65819ea00
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e65819ea00
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9101e7e98400
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f51b8dce00
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9101e7e98400
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df4c8b0000
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df4c8b0000
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f51b8dce00
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9101e7e98400
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f51b8dce00
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9101e7e98400
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f51b8dce00
Apr 22 17:14:33 fir-io2-s2 kernel: LNet: Service thread pid 84799 completed after 236.15s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df4c8b0000
Apr 22 17:14:33 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df4c8b0000
Apr 22 17:14:35 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d2cab30200
Apr 22 17:14:35 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f72661c800
Apr 22 17:14:35 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e343f49600
Apr 22 17:14:48 fir-io2-s2 kernel: LustreError: 137-5: fir-OST000e_UUID: not available for connect from 10.8.17.25@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 17:14:48 fir-io2-s2 kernel: LustreError: Skipped 1355 previous similar messages
Apr 22 17:14:51 fir-io2-s2 kernel: LNet: Service thread pid 84826 completed after 241.37s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Apr 22 17:14:51 fir-io2-s2 kernel: LNet: Skipped 5 previous similar messages
Apr 22 17:14:56 fir-io2-s2 kernel: LustreError: 109222:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.1.10@o2ib6 arrived at 1555978495 with bad export cookie 5677255811366074165
Apr 22 17:14:56 fir-io2-s2 kernel: LustreError: 109222:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 4 previous similar messages
Apr 22 17:15:10 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fda2969600
Apr 22 17:15:22 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3acffc600
Apr 22 17:15:31 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e94cc4f200
Apr 22 17:15:31 fir-io2-s2 kernel: Lustre: fir-OST0015: Bulk IO write error with 4fd3697b-8ac3-d03c-d547-c2a2aae5b292 (at 10.8.28.8@o2ib6), client will retry: rc = -110
Apr 22 17:15:31 fir-io2-s2 kernel: Lustre: Skipped 10 previous similar messages
Apr 22 17:15:35 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f484f91400
Apr 22 17:15:37 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e01bbb0000
Apr 22 17:15:37 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e01bbb0000
Apr 22 17:15:37 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e01bbb0000
Apr 22 17:15:37 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e01bbb0000
Apr 22 17:15:44 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9108ee6f8200
Apr 22 17:15:44 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9108ee6f8200
Apr 22 17:15:44 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9108ee6f8200
Apr 22 17:16:02 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:02 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:02 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:02 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:02 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:02 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:02 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:02 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:02 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:02 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:02 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:05 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 172s: evicting client at 10.8.12.13@o2ib6  ns: filter-fir-OST000f_UUID lock: ffff90d89f9a3cc0/0x4ec9aa14eca5d4f9 lrc: 3/0,0 mode: PR/PR res: [0xac0000402:0x571f1c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400010020 nid: 10.8.12.13@o2ib6 remote: 0x8c517e1b19b9bd60 expref: 1928 pid: 83785 timeout: 24737 lvb_type: 1
Apr 22 17:16:05 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e4202f6200
Apr 22 17:16:05 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 15 previous similar messages
Apr 22 17:16:11 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e92a50cc00
Apr 22 17:16:17 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1f5258000
Apr 22 17:16:17 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1f5258000
Apr 22 17:16:17 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1f5258000
Apr 22 17:16:17 fir-io2-s2 kernel: LustreError: 85105:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff9100e50a1450 x1631531243303760/t0(0) o4->f3bba4e8-9568-4001-6257-88537741a8c9@10.8.29.3@o2ib6:634/0 lens 488/448 e 1 to 0 dl 1555978624 ref 1 fl Interpret:/2/0 rc 0/0
Apr 22 17:16:17 fir-io2-s2 kernel: LustreError: 85105:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 11 previous similar messages
Apr 22 17:16:18 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e92a50cc00
Apr 22 17:16:23 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:23 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:23 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:23 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:23 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d89eb45400
Apr 22 17:16:28 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d577076a00
Apr 22 17:16:29 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7d7847000
Apr 22 17:16:29 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7b3546600
Apr 22 17:16:37 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d577076a00
Apr 22 17:16:44 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d8b9e000
Apr 22 17:16:50 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa1ce0aa00
Apr 22 17:16:51 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d577076a00
Apr 22 17:16:52 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa1ce0aa00
Apr 22 17:16:53 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff6e29b000
Apr 22 17:16:53 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff6e29b000
Apr 22 17:16:53 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff6e29b000
Apr 22 17:16:53 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff6e29b000
Apr 22 17:16:53 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:16:53 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:16:53 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:16:53 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:16:53 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:16:53 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:16:53 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:16:53 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:16:59 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa1ce0aa00
Apr 22 17:17:03 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.8.17.25@o2ib6  ns: filter-fir-OST000f_UUID lock: ffff90f221fe0fc0/0x4ec9aa14da8579b1 lrc: 3/0,0 mode: PR/PR res: [0xac0000402:0x574f92:0x0].0x0 rrc: 15 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400010020 nid: 10.8.17.25@o2ib6 remote: 0x7d848fc790d024cd expref: 946 pid: 83747 timeout: 24795 lvb_type: 1
Apr 22 17:17:03 fir-io2-s2 kernel: LustreError: 45146:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff9104a36e6000 x1631536998706800/t0(0) o105->fir-OST000d@10.8.29.4@o2ib6:15/16 lens 360/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1
Apr 22 17:17:03 fir-io2-s2 kernel: LustreError: 45146:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
Apr 22 17:17:05 fir-io2-s2 kernel: LNetError: 81270:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 17:17:05 fir-io2-s2 kernel: LNetError: 81270:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 121 previous similar messages
Apr 22 17:17:05 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff6e29b000
Apr 22 17:17:05 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:05 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:05 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:05 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:05 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:05 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:17:05 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:17:05 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:17:05 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff6e29b000
Apr 22 17:17:15 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f283947000
Apr 22 17:17:16 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fb78c8b000
Apr 22 17:17:16 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e794018c00
Apr 22 17:17:20 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1440a8a00
Apr 22 17:17:20 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff6e29b000
Apr 22 17:17:20 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff6e29b000
Apr 22 17:17:20 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:20 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:20 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:20 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:17:20 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:17:20 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:17:20 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:17:20 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:17:20 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff6e29b000
Apr 22 17:17:22 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e430071000
Apr 22 17:17:22 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e430071000
Apr 22 17:17:22 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dc02b37400
Apr 22 17:17:22 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f671ac3800
Apr 22 17:17:22 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dc02b37400
Apr 22 17:17:22 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e398ce0400
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0457e0400
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0457e0400
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0457e0400
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff6e29b000
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff6e29b000
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff6e29b000
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c84e00
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67d20ee00
Apr 22 17:17:26 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0457e0400
Apr 22 17:17:30 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e71f2a3c00
Apr 22 17:17:30 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e71f2a3c00
Apr 22 17:17:30 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e71f2a3c00
Apr 22 17:17:30 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e430071000
Apr 22 17:17:30 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e398ce0400
Apr 22 17:17:30 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f671ac3800
Apr 22 17:17:30 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dc02b37400
Apr 22 17:17:30 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f671ac3800
Apr 22 17:17:33 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff2b411400
Apr 22 17:17:42 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e9cbc14800
Apr 22 17:17:44 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f762028600
Apr 22 17:17:45 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f4b8e5da00
Apr 22 17:17:45 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8b5df6800
Apr 22 17:17:47 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fb78c8ca00
Apr 22 17:17:47 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efeac2ee00
Apr 22 17:17:47 fir-io2-s2 kernel: Lustre: fir-OST0011: Bulk IO read error with 4a5b9079-3e8c-70ae-a88f-6098c9496628 (at 10.8.28.4@o2ib6), client will retry: rc -110
Apr 22 17:17:47 fir-io2-s2 kernel: Lustre: Skipped 34 previous similar messages
Apr 22 17:17:47 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e53281a200
Apr 22 17:17:47 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7d0322600
Apr 22 17:17:47 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa361ef000
Apr 22 17:17:47 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fb2a94a400
Apr 22 17:17:47 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e782f95a00
Apr 22 17:17:48 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90faeb791400
Apr 22 17:17:48 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f30c95ba00
Apr 22 17:17:50 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e219beb200
Apr 22 17:17:51 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddcc342e00
Apr 22 17:17:52 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8b5df6800
Apr 22 17:17:53 fir-io2-s2 kernel: LustreError: 85231:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff90f115549c50 x1631545734085472/t0(0) o3->8776df39-692a-3df9-0874-72e441440742@10.8.18.22@o2ib6:42/0 lens 488/440 e 0 to 0 dl 1555978787 ref 1 fl Interpret:/2/0 rc 0/0
Apr 22 17:17:53 fir-io2-s2 kernel: LustreError: 85231:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 21 previous similar messages
Apr 22 17:17:56 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90defb615400
Apr 22 17:17:57 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db58b53200
Apr 22 17:17:58 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.8.12.17@o2ib6  ns: filter-fir-OST000d_UUID lock: ffff90d557aa57c0/0x4ec9aa14ece43dd1 lrc: 3/0,0 mode: PR/PR res: [0x300000400:0x574d86:0x0].0x0 rrc: 13 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400010020 nid: 10.8.12.17@o2ib6 remote: 0x8200f803b72d1558 expref: 918 pid: 84884 timeout: 24850 lvb_type: 1
Apr 22 17:17:58 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages
Apr 22 17:18:00 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dd103f6800
Apr 22 17:18:03 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f414a33200
Apr 22 17:18:03 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ed9dfcfa00
Apr 22 17:18:07 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e9aaf8f000
Apr 22 17:18:07 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e8d7e55800
Apr 22 17:18:07 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eaedfb7400
Apr 22 17:18:07 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ee445dcc00
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f13df34c00
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f13df34c00
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f13df34c00
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f13df34c00
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f13df34c00
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f13df34c00
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e88cd34200
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e88cd34200
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e88cd34200
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e88cd34200
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df93a46e00
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df93a46e00
Apr 22 17:18:08 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df93a46e00
Apr 22 17:18:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d4078ef200
Apr 22 17:18:11 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddbdbefc00
Apr 22 17:18:11 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d429e2da00
Apr 22 17:18:13 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f13df34c00
Apr 22 17:18:13 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f13df34c00
Apr 22 17:18:13 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f13df34c00
Apr 22 17:18:13 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f13df34c00
Apr 22 17:18:13 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e88cd34200
Apr 22 17:18:14 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f06a94dc00
Apr 22 17:18:14 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c87200
Apr 22 17:18:14 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f9f955ca00
Apr 22 17:18:14 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f733b4a000
Apr 22 17:18:14 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f06a94dc00
Apr 22 17:18:16 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e60b4a8c00
Apr 22 17:18:23 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dba69e7400
Apr 22 17:18:25 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f06a94dc00
Apr 22 17:18:25 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d3d236ae00
Apr 22 17:18:27 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e519910600
Apr 22 17:18:28 fir-io2-s2 kernel: Lustre: fir-OST0011: Connection restored to 1c06f372-dead-1042-58da-264d882a9ae5 (at 10.8.17.20@o2ib6)
Apr 22 17:18:28 fir-io2-s2 kernel: Lustre: Skipped 1324 previous similar messages
Apr 22 17:18:28 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dba69e7400
Apr 22 17:18:29 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd29979a00
Apr 22 17:18:29 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd29979a00
Apr 22 17:18:29 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd29979a00
Apr 22 17:18:30 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f06a94dc00
Apr 22 17:18:30 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e69f15da00
Apr 22 17:18:36 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dcc8416200
Apr 22 17:18:44 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e652aa6400
Apr 22 17:18:44 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 158s: evicting client at 10.8.17.24@o2ib6  ns: filter-fir-OST000d_UUID lock: ffff90d825a386c0/0x4ec9aa14ecc76437 lrc: 3/0,0 mode: PR/PR res: [0x300000400:0x5741f4:0x0].0x0 rrc: 8 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.8.17.24@o2ib6 remote: 0xa0384b38e042f06e expref: 856 pid: 84983 timeout: 24896 lvb_type: 1
Apr 22 17:18:44 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages
Apr 22 17:19:04 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e650eaf600
Apr 22 17:19:05 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dd97992200
Apr 22 17:19:06 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff91012ab65400
Apr 22 17:19:16 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9100332b0200
Apr 22 17:19:16 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ed5cbaee00
Apr 22 17:19:16 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff91012ab65400
Apr 22 17:19:21 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f41fc5ce00
Apr 22 17:19:22 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f63429b200
Apr 22 17:19:23 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff91012ab65400
Apr 22 17:19:27 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dbb82e3800
Apr 22 17:19:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ee6ec07a00
Apr 22 17:19:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ee6ec07a00
Apr 22 17:19:33 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ecde689e00
Apr 22 17:19:33 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd309c7c00
Apr 22 17:19:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ecde689e00
Apr 22 17:19:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd309c7c00
Apr 22 17:19:34 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f985845c00
Apr 22 17:19:34 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f0dced5e00
Apr 22 17:19:34 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e548efa000
Apr 22 17:19:38 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e27cbe7200
Apr 22 17:19:38 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ee6ec07a00
Apr 22 17:19:38 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ee6ec07a00
Apr 22 17:19:38 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd309c7c00
Apr 22 17:19:38 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ecde689e00
Apr 22 17:19:38 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ecde689e00
Apr 22 17:19:38 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd309c7c00
Apr 22 17:19:43 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f178503400
Apr 22 17:19:45 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efde6a3200
Apr 22 17:19:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efde6a3200
Apr 22 17:19:56 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:19:56 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:19:56 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:19:56 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:19:56 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:19:56 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:00 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef791a9400
Apr 22 17:20:00 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef791a9400
Apr 22 17:20:00 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef791a9400
Apr 22 17:20:00 fir-io2-s2 kernel: Lustre: fir-OST000f: Bulk IO write error with aba5d4eb-e07c-9b0f-6ab5-7f97caf38a26 (at 10.8.16.4@o2ib6), client will retry: rc = -110
Apr 22 17:20:00 fir-io2-s2 kernel: Lustre: Skipped 28 previous similar messages
Apr 22 17:20:02 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e48bc1fc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e563c17600
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eeb2d7e000
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:10 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f486bcdc00
Apr 22 17:20:12 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d75f5ff400
Apr 22 17:20:12 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d75f5ff400
Apr 22 17:20:12 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d75f5ff400
Apr 22 17:20:13 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9101f95a5e00
Apr 22 17:20:13 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d646fd6e00
Apr 22 17:20:15 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1c271a000
Apr 22 17:20:15 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f224c81e00
Apr 22 17:20:15 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e6afd44000
Apr 22 17:20:18 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d577074200
Apr 22 17:20:23 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f9706bdc00
Apr 22 17:20:23 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dafe451000
Apr 22 17:20:23 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e453be1200
Apr 22 17:20:26 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec3216ec00
Apr 22 17:20:28 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90edf9fffc00
Apr 22 17:20:28 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eaedfb6c00
Apr 22 17:20:28 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f95c481200
Apr 22 17:20:36 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e140e8be00
Apr 22 17:20:36 fir-io2-s2 kernel: LustreError: 84960:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk READ  req@ffff90e2869d3450 x1631534592123232/t0(0) o3->f2a69869-02b0-7522-b419-5cd488a596b0@10.8.18.12@o2ib6:202/0 lens 488/440 e 1 to 0 dl 1555978947 ref 1 fl Interpret:/2/0 rc 0/0
Apr 22 17:20:36 fir-io2-s2 kernel: LustreError: 84960:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 89 previous similar messages
Apr 22 17:20:36 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ece6baf600
Apr 22 17:20:36 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb16803600
Apr 22 17:20:36 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb16803600
Apr 22 17:20:36 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7cd49e200
Apr 22 17:20:36 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ea48269200
Apr 22 17:20:37 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7eb49f200
Apr 22 17:20:37 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ea687f5200
Apr 22 17:20:37 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f412b14400
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507831600
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507831600
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507831600
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507831600
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:39 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:43 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec7f728400
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e420e2a800
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e420e2a800
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e420e2a800
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e420e2a800
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507831600
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507831600
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507831600
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507831600
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507831600
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507831600
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8d561be00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa668b6c00
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507831600
Apr 22 17:20:46 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e507831600
Apr 22 17:20:50 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fbb6fcc800
Apr 22 17:20:55 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da09c86a00
Apr 22 17:20:55 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e31a0a1600
Apr 22 17:21:02 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ea3d8f5e00
Apr 22 17:21:02 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f43593a400
Apr 22 17:21:03 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e31a0a1600
Apr 22 17:21:03 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f71dfb0a00
Apr 22 17:21:04 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3d2b51000
Apr 22 17:21:04 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3d2b51000
Apr 22 17:21:04 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3d2b51000
Apr 22 17:21:04 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3d2b51000
Apr 22 17:21:05 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fdcae1aa00
Apr 22 17:21:10 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3e4a48a00
Apr 22 17:21:16 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efeac29800
Apr 22 17:21:16 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efeac29800
Apr 22 17:21:16 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efeac29800
Apr 22 17:21:16 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efeac29800
Apr 22 17:21:16 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efeac29800
Apr 22 17:21:16 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efeac29800
Apr 22 17:21:16 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efeac29800
Apr 22 17:21:16 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efeac29800
Apr 22 17:21:16 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ed12e8ea00
Apr 22 17:21:16 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fe64199a00
Apr 22 17:21:16 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ed12e8ea00
Apr 22 17:21:23 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fdb80b2c00
Apr 22 17:21:23 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d908d59200
Apr 22 17:21:26 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7df75ba00
Apr 22 17:21:26 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90def31c2600
Apr 22 17:21:26 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:26 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:26 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:26 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:26 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:27 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:27 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:27 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:27 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:27 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:27 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:27 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:29 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fcbfbd5000
Apr 22 17:21:30 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:30 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:30 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:30 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:30 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:30 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:30 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:30 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:30 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:30 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:30 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8f011e800
Apr 22 17:21:39 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90edd0a4b400
Apr 22 17:21:40 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:40 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:40 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:40 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:40 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e645fa4e00
Apr 22 17:21:40 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:40 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:40 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:40 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:40 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8adb69600
Apr 22 17:21:48 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd9493ca00
Apr 22 17:21:50 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de6d4c2200
Apr 22 17:21:50 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e4add69600
Apr 22 17:21:54 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dab33df000
Apr 22 17:21:54 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f9d28cf400
Apr 22 17:21:55 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de8f301a00
Apr 22 17:22:13 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f190676800
Apr 22 17:22:18 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fc2efaf200
Apr 22 17:22:28 fir-io2-s2 kernel: Lustre: fir-OST0015: haven't heard from client 7bc968ec-0af0-1f9a-1f8e-b6540b9ff2ce (at 10.8.29.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90f2b8f9b400, cur 1555978948 expire 1555978798 last 1555978721
Apr 22 17:22:28 fir-io2-s2 kernel: Lustre: Skipped 60 previous similar messages
Apr 22 17:22:38 fir-io2-s2 kernel: LustreError: 85194:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(48982)  req@ffff90fed76b6450 x1631320591904256/t0(0) o4->ccc8b821-13fa-5ca1-49da-be1a6cb0e16f@10.8.30.27@o2ib6:326/0 lens 488/448 e 0 to 0 dl 1555979071 ref 1 fl Interpret:/2/0 rc 0/0
Apr 22 17:22:39 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebc871d400
Apr 22 17:22:39 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebc871d400
Apr 22 17:22:39 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebc871d400
Apr 22 17:22:39 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebc871d400
Apr 22 17:22:41 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f9d01c3c00
Apr 22 17:22:41 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dc0152d000
Apr 22 17:22:43 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ece4cbde00
Apr 22 17:22:43 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3dff28800
Apr 22 17:22:43 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e33605e000
Apr 22 17:22:43 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3dff28800
Apr 22 17:22:43 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3dff28800
Apr 22 17:22:43 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3dff28800
Apr 22 17:22:43 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7efc15c00
Apr 22 17:22:43 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7efc15c00
Apr 22 17:22:43 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7efc15c00
Apr 22 17:22:43 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7efc15c00
Apr 22 17:22:45 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f948796000
Apr 22 17:22:45 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e758ea5200
Apr 22 17:22:45 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f147294600
Apr 22 17:22:45 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3c8591e00
Apr 22 17:23:03 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f969af1000
Apr 22 17:23:03 fir-io2-s2 kernel: LustreError: 84976:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91012d487850 x1631322761179504/t0(0) o4->978af126-4ab9-9875-237d-0d9b43dd32e8@10.8.10.21@o2ib6:266/0 lens 488/448 e 1 to 0 dl 1555979011 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 17:23:03 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efa9e9ec00
Apr 22 17:23:03 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efa9e9ec00
Apr 22 17:23:06 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90edf2397600
Apr 22 17:23:06 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90edf2397600
Apr 22 17:24:22 fir-io2-s2 kernel: Lustre: fir-OST0013: Client d3013375-2e90-b76e-c4d8-76867f2b4a32 (at 10.8.2.20@o2ib6) reconnecting
Apr 22 17:24:22 fir-io2-s2 kernel: Lustre: Skipped 573 previous similar messages
Apr 22 17:24:41 fir-io2-s2 kernel: Lustre: 84881:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555979074/real 0]  req@ffff90fb7eb49500 x1631537003788912/t0(0) o104->fir-OST0011@10.8.8.29@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555979081 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Apr 22 17:24:41 fir-io2-s2 kernel: Lustre: 84881:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 182 previous similar messages
Apr 22 17:25:11 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f81e34e800
Apr 22 17:25:15 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef0a314a00
Apr 22 17:25:16 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e11fa97800
Apr 22 17:28:29 fir-io2-s2 kernel: LustreError: 137-5: fir-OST000c_UUID: not available for connect from 10.8.27.25@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 17:28:29 fir-io2-s2 kernel: LustreError: Skipped 422 previous similar messages
Apr 22 17:28:54 fir-io2-s2 kernel: Lustre: fir-OST000d: Connection restored to a4daaf47-6ec9-4753-388e-0d0b7a7f70d6 (at 10.8.27.25@o2ib6)
Apr 22 17:28:54 fir-io2-s2 kernel: Lustre: Skipped 303 previous similar messages
Apr 22 17:30:24 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.9.108.57@o2ib4  ns: filter-fir-OST0011_UUID lock: ffff90de47f586c0/0x4ec9aa14f0667dd7 lrc: 3/0,0 mode: PW/PW res: [0x2a8af2:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400020020 nid: 10.9.108.57@o2ib4 remote: 0xd0cbc03593f2af48 expref: 23 pid: 84954 timeout: 25596 lvb_type: 0
Apr 22 17:30:24 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages
Apr 22 17:33:37 fir-io2-s2 kernel: Lustre: fir-OST000d: haven't heard from client 1699114c-d7be-fde3-c1c0-12a75a1676f5 (at 10.8.27.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91121f02b800, cur 1555979617 expire 1555979467 last 1555979390
Apr 22 17:33:37 fir-io2-s2 kernel: Lustre: Skipped 52 previous similar messages
Apr 22 17:33:57 fir-io2-s2 kernel: LustreError: 84853:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.25@o2ib6) failed to reply to blocking AST (req@ffff90f779550900 x1631537003805872 status 0 rc -110), evict it ns: filter-fir-OST0013_UUID lock: ffff90f0fe346300/0x4ec9aa14ed99bf84 lrc: 4/0,0 mode: PR/PR res: [0x600000402:0x927a41:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.8.27.25@o2ib6 remote: 0xca2cb7566c72ef94 expref: 312963 pid: 83760 timeout: 25951 lvb_type: 1
Apr 22 17:33:57 fir-io2-s2 kernel: LustreError: 138-a: fir-OST0013: A client on nid 10.8.27.25@o2ib6 was evicted due to a lock blocking callback time out: rc -110
Apr 22 17:33:57 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 155s: evicting client at 10.8.27.25@o2ib6  ns: filter-fir-OST0013_UUID lock: ffff90f0fe346300/0x4ec9aa14ed99bf84 lrc: 3/0,0 mode: PR/PR res: [0x600000402:0x927a41:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.8.27.25@o2ib6 remote: 0xca2cb7566c72ef94 expref: 312964 pid: 83760 timeout: 0 lvb_type: 1
Apr 22 17:39:14 fir-io2-s2 kernel: Lustre: fir-OST000d: Connection restored to f3b5dfe4-1917-2d20-fd73-c3a1a450a23e (at 10.8.18.1@o2ib6)
Apr 22 17:39:14 fir-io2-s2 kernel: Lustre: Skipped 160 previous similar messages
Apr 22 17:40:03 fir-io2-s2 kernel: Lustre: DEBUG MARKER: Mon Apr 22 17:40:03 2019
Apr 22 17:44:42 fir-io2-s2 kernel: Lustre: fir-OST000d: haven't heard from client 21eb1043-77e2-ccc0-0725-7be9fa6337bc (at 10.9.106.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91121f15e400, cur 1555980282 expire 1555980132 last 1555980055
Apr 22 17:44:42 fir-io2-s2 kernel: Lustre: Skipped 17 previous similar messages
Apr 22 17:54:14 fir-io2-s2 kernel: Lustre: fir-OST0015: Connection restored to a4daaf47-6ec9-4753-388e-0d0b7a7f70d6 (at 10.8.27.25@o2ib6)
Apr 22 17:54:14 fir-io2-s2 kernel: Lustre: fir-OST0011: Connection restored to a4daaf47-6ec9-4753-388e-0d0b7a7f70d6 (at 10.8.27.25@o2ib6)
Apr 22 17:54:14 fir-io2-s2 kernel: Lustre: Skipped 20 previous similar messages
Apr 22 17:54:14 fir-io2-s2 kernel: Lustre: Skipped 4 previous similar messages
Apr 22 17:57:36 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.9.104.2@o2ib4  ns: filter-fir-OST0013_UUID lock: ffff90e23c2ee780/0x4ec9aa14f066fed9 lrc: 3/0,0 mode: PW/PW res: [0x2a8bbe:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400020020 nid: 10.9.104.2@o2ib4 remote: 0x85bbe72d55ff1102 expref: 8 pid: 83771 timeout: 27228 lvb_type: 0
Apr 22 17:58:26 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.8.11.26@o2ib6  ns: filter-fir-OST0011_UUID lock: ffff911266f8bf00/0x4ec9aa14f06729c0 lrc: 3/0,0 mode: PW/PW res: [0x680000400:0x56e61c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x60000400020020 nid: 10.8.11.26@o2ib6 remote: 0xa234b98ee14e9728 expref: 6 pid: 85005 timeout: 27278 lvb_type: 0
Apr 22 18:00:04 fir-io2-s2 kernel: Lustre: fir-OST0013: haven't heard from client d0508a0a-2aa4-bc4f-972b-2a2c8c0a4472 (at 10.9.103.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91126424c800, cur 1555981204 expire 1555981054 last 1555980977
Apr 22 18:00:04 fir-io2-s2 kernel: Lustre: Skipped 11 previous similar messages
Apr 22 18:01:42 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.9.108.57@o2ib4  ns: filter-fir-OST0015_UUID lock: ffff9101ad674c80/0x4ec9aa14f06711ef lrc: 3/0,0 mode: PW/PW res: [0x2a8d71:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x60000400020020 nid: 10.9.108.57@o2ib4 remote: 0xd0cbc0359410f7ba expref: 27 pid: 82391 timeout: 27474 lvb_type: 0
Apr 22 18:02:52 fir-io2-s2 kernel: LustreError: 85086:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000d: cli 1b540135-ca45-f45a-a248-eb5cb55b2913 claims 4218880 GRANT, real grant 0
Apr 22 18:03:46 fir-io2-s2 kernel: LustreError: 82406:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000d: cli 1b540135-ca45-f45a-a248-eb5cb55b2913 claims 4218880 GRANT, real grant 0
Apr 22 18:03:46 fir-io2-s2 kernel: LustreError: 82406:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 1 previous similar message
Apr 22 18:04:19 fir-io2-s2 kernel: LustreError: 85002:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000d: cli 1b540135-ca45-f45a-a248-eb5cb55b2913 claims 3706880 GRANT, real grant 0
Apr 22 18:04:19 fir-io2-s2 kernel: LustreError: 85002:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 10 previous similar messages
Apr 22 18:04:41 fir-io2-s2 kernel: LustreError: 85125:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000d: cli 1b540135-ca45-f45a-a248-eb5cb55b2913 claims 4218880 GRANT, real grant 0
Apr 22 18:07:13 fir-io2-s2 kernel: Lustre: fir-OST000d: Connection restored to 21eb1043-77e2-ccc0-0725-7be9fa6337bc (at 10.9.106.17@o2ib4)
Apr 22 18:07:13 fir-io2-s2 kernel: Lustre: Skipped 64 previous similar messages
Apr 22 18:07:20 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.8.29.3@o2ib6  ns: filter-fir-OST0011_UUID lock: ffff90e1b7b9ba80/0x4ec9aa14f0671ea6 lrc: 3/0,0 mode: PW/PW res: [0x27c2ce:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400020020 nid: 10.8.29.3@o2ib6 remote: 0x73c64d7d4870f823 expref: 25 pid: 83775 timeout: 27812 lvb_type: 0
Apr 22 18:07:20 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message
Apr 22 18:07:24 fir-io2-s2 kernel: LustreError: 99409:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.29.3@o2ib6 arrived at 1555981643 with bad export cookie 5677255811366070896
Apr 22 18:07:24 fir-io2-s2 kernel: LustreError: 85127:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000d: cli 1b540135-ca45-f45a-a248-eb5cb55b2913 claims 4218880 GRANT, real grant 0
Apr 22 18:07:24 fir-io2-s2 kernel: LustreError: 85127:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 10 previous similar messages
Apr 22 18:09:39 fir-io2-s2 kernel: LustreError: 85197:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000d: cli 1b540135-ca45-f45a-a248-eb5cb55b2913 claims 4218880 GRANT, real grant 0
Apr 22 18:09:39 fir-io2-s2 kernel: LustreError: 85197:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 3 previous similar messages
Apr 22 18:09:52 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.9.108.57@o2ib4  ns: filter-fir-OST0011_UUID lock: ffff90da80740d80/0x4ec9aa14f0677420 lrc: 3/0,0 mode: PW/PW res: [0x2a8af2:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400020020 nid: 10.9.108.57@o2ib4 remote: 0xd0cbc0359410f9b9 expref: 6 pid: 84818 timeout: 27964 lvb_type: 0
Apr 22 18:10:07 fir-io2-s2 kernel: LustreError: 35438:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000d: cli 1b540135-ca45-f45a-a248-eb5cb55b2913 claims 4218880 GRANT, real grant 0
Apr 22 18:10:07 fir-io2-s2 kernel: LustreError: 35438:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 3 previous similar messages
Apr 22 18:10:40 fir-io2-s2 kernel: LustreError: 85185:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000d: cli 1b540135-ca45-f45a-a248-eb5cb55b2913 claims 3706880 GRANT, real grant 0
Apr 22 18:10:40 fir-io2-s2 kernel: LustreError: 85185:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 10 previous similar messages
Apr 22 18:12:23 fir-io2-s2 kernel: LustreError: 85096:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000d: cli 1b540135-ca45-f45a-a248-eb5cb55b2913 claims 4218880 GRANT, real grant 0
Apr 22 18:12:23 fir-io2-s2 kernel: LustreError: 85096:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 23 previous similar messages
Apr 22 18:13:45 fir-io2-s2 kernel: Lustre: fir-OST000d: haven't heard from client feb98fcf-e5cd-05d5-3c3f-508f3fbfb9c4 (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91121c614c00, cur 1555982025 expire 1555981875 last 1555981798
Apr 22 18:13:45 fir-io2-s2 kernel: Lustre: Skipped 19 previous similar messages
Apr 22 18:15:20 fir-io2-s2 kernel: LustreError: 85251:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000d: cli 1b540135-ca45-f45a-a248-eb5cb55b2913 claims 4218880 GRANT, real grant 0
Apr 22 18:15:20 fir-io2-s2 kernel: LustreError: 85251:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 18 previous similar messages
Apr 22 18:19:57 fir-io2-s2 kernel: Lustre: fir-OST000d: Connection restored to 06c46967-6dc5-73fe-bdf7-9ebb7f764452 (at 10.8.27.23@o2ib6)
Apr 22 18:19:57 fir-io2-s2 kernel: Lustre: Skipped 21 previous similar messages
Apr 22 18:20:59 fir-io2-s2 kernel: LustreError: 85197:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000d: cli 1b540135-ca45-f45a-a248-eb5cb55b2913 claims 4218880 GRANT, real grant 0
Apr 22 18:20:59 fir-io2-s2 kernel: LustreError: 85197:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 33 previous similar messages
Apr 22 18:24:59 fir-io2-s2 kernel: Lustre: fir-OST000f: haven't heard from client 9773eb26-ebf6-6918-5851-ee91be7eff02 (at 10.9.104.58@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91120252b400, cur 1555982699 expire 1555982549 last 1555982472
Apr 22 18:24:59 fir-io2-s2 kernel: Lustre: Skipped 23 previous similar messages
Apr 22 18:28:03 fir-io2-s2 kernel: Lustre: fir-OST0013: Client da84e749-aae7-e201-4af5-a27465c5b762 (at 10.8.0.65@o2ib6) reconnecting
Apr 22 18:28:03 fir-io2-s2 kernel: Lustre: Skipped 148 previous similar messages
Apr 22 18:29:53 fir-io2-s2 kernel: LustreError: 84952:0:(tgt_grant.c:742:tgt_grant_check()) fir-OST000d: cli 1b540135-ca45-f45a-a248-eb5cb55b2913 claims 4218880 GRANT, real grant 0
Apr 22 18:29:53 fir-io2-s2 kernel: LustreError: 84952:0:(tgt_grant.c:742:tgt_grant_check()) Skipped 51 previous similar messages
Apr 22 18:31:59 fir-io2-s2 kernel: Lustre: fir-OST000d: Connection restored to c366c373-7a23-824e-f63d-44affd1c5112 (at 10.9.104.35@o2ib4)
Apr 22 18:31:59 fir-io2-s2 kernel: Lustre: Skipped 24 previous similar messages
Apr 22 18:35:54 fir-io2-s2 kernel: Lustre: fir-OST0013: Client d8e463c2-b8d9-0853-9f4e-aeede1da06d9 (at 10.8.13.14@o2ib6) reconnecting
Apr 22 18:35:54 fir-io2-s2 kernel: Lustre: Skipped 1 previous similar message
Apr 22 18:36:02 fir-io2-s2 kernel: Lustre: 84771:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555983355/real 0]  req@ffff90f0079f8000 x1631537030685984/t0(0) o104->fir-OST0017@10.8.2.24@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555983362 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Apr 22 18:36:02 fir-io2-s2 kernel: Lustre: 84771:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 53 previous similar messages
Apr 22 18:36:04 fir-io2-s2 kernel: LustreError: 85128:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff90e2205eb450 x1631532760416720/t0(0) o3->fe1114de-6afc-56c5-59b3-ce46a56cb035@10.8.2.15@o2ib6:128/0 lens 488/440 e 0 to 0 dl 1555983403 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:36:04 fir-io2-s2 kernel: LustreError: 85128:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 12 previous similar messages
Apr 22 18:36:09 fir-io2-s2 kernel: LNetError: 81278:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 18:36:09 fir-io2-s2 kernel: LNetError: 81278:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 365 previous similar messages
Apr 22 18:36:09 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90feae3f3e00
Apr 22 18:36:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90feae3f3e00
Apr 22 18:36:10 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df75d9ea00
Apr 22 18:36:10 fir-io2-s2 kernel: Lustre: fir-OST000f: Bulk IO read error with fe1114de-6afc-56c5-59b3-ce46a56cb035 (at 10.8.2.15@o2ib6), client will retry: rc -110
Apr 22 18:36:10 fir-io2-s2 kernel: Lustre: Skipped 90 previous similar messages
Apr 22 18:36:11 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e5b388ce00
Apr 22 18:36:11 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e5b388ce00
Apr 22 18:36:11 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e5b388ce00
Apr 22 18:36:11 fir-io2-s2 kernel: Lustre: fir-OST0017: Bulk IO write error with fe1114de-6afc-56c5-59b3-ce46a56cb035 (at 10.8.2.15@o2ib6), client will retry: rc = -110
Apr 22 18:36:11 fir-io2-s2 kernel: Lustre: Skipped 31 previous similar messages
Apr 22 18:36:22 fir-io2-s2 kernel: LustreError: 137-5: fir-OST000e_UUID: not available for connect from 10.8.29.4@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 18:36:22 fir-io2-s2 kernel: LustreError: 137-5: fir-OST000c_UUID: not available for connect from 10.8.29.4@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 18:36:22 fir-io2-s2 kernel: LustreError: Skipped 9 previous similar messages
Apr 22 18:36:44 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ed97695a00
Apr 22 18:37:38 fir-io2-s2 kernel: LustreError: 137-5: fir-OST0016_UUID: not available for connect from 10.8.9.1@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 18:37:38 fir-io2-s2 kernel: LustreError: Skipped 678 previous similar messages
Apr 22 18:37:52 fir-io2-s2 kernel: Lustre: 83755:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555983465/real 0]  req@ffff90f29e3cdd00 x1631537032267456/t0(0) o104->fir-OST0015@10.8.0.66@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555983472 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Apr 22 18:37:52 fir-io2-s2 kernel: Lustre: 83755:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Apr 22 18:38:24 fir-io2-s2 kernel: Lustre: fir-OST0017: Client 27d764b1-f6cd-d678-112d-32dca2473c6a (at 10.8.26.24@o2ib6) reconnecting
Apr 22 18:38:24 fir-io2-s2 kernel: Lustre: Skipped 1834 previous similar messages
Apr 22 18:38:53 fir-io2-s2 kernel: Lustre: fir-OST0015: haven't heard from client 8abac2b4-4ab8-a3ac-d5a6-c3490c0dd72f (at 10.9.106.26@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90f1ac205c00, cur 1555983533 expire 1555983383 last 1555983306
Apr 22 18:38:53 fir-io2-s2 kernel: Lustre: Skipped 53 previous similar messages
Apr 22 18:39:48 fir-io2-s2 kernel: LustreError: 85002:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff90e27faafc50 x1631532760500096/t0(0) o4->fe1114de-6afc-56c5-59b3-ce46a56cb035@10.8.2.15@o2ib6:345/0 lens 488/448 e 0 to 0 dl 1555983620 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:39:48 fir-io2-s2 kernel: LustreError: 85002:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 2 previous similar messages
Apr 22 18:40:02 fir-io2-s2 kernel: LNetError: 81277:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 18:40:02 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e9b668fc00
Apr 22 18:40:02 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e9b668fc00
Apr 22 18:40:02 fir-io2-s2 kernel: LNetError: 81277:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 8 previous similar messages
Apr 22 18:40:02 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e9b668fc00
Apr 22 18:40:02 fir-io2-s2 kernel: Lustre: fir-OST000f: Bulk IO write error with fe1114de-6afc-56c5-59b3-ce46a56cb035 (at 10.8.2.15@o2ib6), client will retry: rc = -110
Apr 22 18:40:03 fir-io2-s2 kernel: LustreError: 85214:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff90f29dac0450 x1631364995904832/t0(0) o3->707fc21c-7c6f-25fd-a7db-dd646b4ab0c6@10.8.10.10@o2ib6:339/0 lens 488/440 e 1 to 0 dl 1555983614 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:40:03 fir-io2-s2 kernel: Lustre: fir-OST0013: Bulk IO read error with 707fc21c-7c6f-25fd-a7db-dd646b4ab0c6 (at 10.8.10.10@o2ib6), client will retry: rc -107
Apr 22 18:40:03 fir-io2-s2 kernel: Lustre: Skipped 2 previous similar messages
Apr 22 18:40:06 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f72b393c00
Apr 22 18:40:06 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f72b393c00
Apr 22 18:40:07 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebacdb0e00
Apr 22 18:40:09 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d743a27800
Apr 22 18:40:09 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d743a27800
Apr 22 18:40:09 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d743a27800
Apr 22 18:40:09 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d743a27800
Apr 22 18:40:09 fir-io2-s2 kernel: LustreError: 85255:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff91012d53f450 x1631287973763392/t0(0) o4->1113817e-df5c-b4f9-94ce-f33b62a1b499@10.8.8.26@o2ib6:345/0 lens 488/448 e 0 to 0 dl 1555983620 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:40:09 fir-io2-s2 kernel: LustreError: 85255:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 53 previous similar messages
Apr 22 18:40:10 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3856f5600
Apr 22 18:40:10 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3856f5600
Apr 22 18:40:10 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3856f5600
Apr 22 18:40:10 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3856f5600
Apr 22 18:40:15 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d81fe1b800
Apr 22 18:40:15 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e605c15e00
Apr 22 18:40:15 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e605c15e00
Apr 22 18:40:16 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d41a063600
Apr 22 18:40:20 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f253761400
Apr 22 18:40:20 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f253761400
Apr 22 18:40:20 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f253761400
Apr 22 18:40:20 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f253761400
Apr 22 18:40:20 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f253761400
Apr 22 18:40:20 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f253761400
Apr 22 18:40:20 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f253761400
Apr 22 18:40:20 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f253761400
Apr 22 18:40:21 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f253761400
Apr 22 18:40:21 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f253761400
Apr 22 18:40:21 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f253761400
Apr 22 18:40:21 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f253761400
Apr 22 18:40:21 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e9b7587800
Apr 22 18:40:22 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff7392a800
Apr 22 18:40:22 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff7392a800
Apr 22 18:40:22 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff7392a800
Apr 22 18:40:22 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff7392a800
Apr 22 18:40:23 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ea31534a00
Apr 22 18:40:23 fir-io2-s2 kernel: LustreError: 85411:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff90f11e60b050 x1631534524163232/t0(0) o3->4fd3697b-8ac3-d03c-d547-c2a2aae5b292@10.8.28.8@o2ib6:364/0 lens 488/440 e 2 to 0 dl 1555983639 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:40:24 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ea8f50e600
Apr 22 18:40:26 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fc305af200
Apr 22 18:40:27 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0c1b0b000
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 85334:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff90f19af13c50 x1631355200338720/t0(0) o3->25e208b5-e2fa-40eb-cac9-4926b51f7128@10.8.12.17@o2ib6:359/0 lens 488/440 e 1 to 0 dl 1555983634 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebacd68c00
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ed07380600
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebacd78000
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f0b05c7200
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8e0a23400
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef9d5f6600
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db9c4e0c00
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e997170800
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e88954ca00
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f526886e00
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ede8bb7c00
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f9931ec800
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff26da8800
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e86b84e400
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ed026dec00
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0e454f400
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0e454f400
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0e454f400
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0e454f400
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0e454f400
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0e454f400
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0e454f400
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0e454f400
Apr 22 18:40:28 fir-io2-s2 kernel: LustreError: 85334:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 6 previous similar messages
Apr 22 18:40:29 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e968e68a00
Apr 22 18:40:30 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddd9102a00
Apr 22 18:40:30 fir-io2-s2 kernel: LustreError: 85336:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff90f219857450 x1631542299573120/t0(0) o3->d63616e9-cace-0db0-1e58-00c83619e62e@10.8.7.24@o2ib6:371/0 lens 488/440 e 2 to 0 dl 1555983646 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:40:30 fir-io2-s2 kernel: LustreError: 85336:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 4 previous similar messages
Apr 22 18:40:31 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efe278d400
Apr 22 18:40:33 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f700fc0a00
Apr 22 18:40:34 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e496bb3000
Apr 22 18:40:34 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebacdb0e00
Apr 22 18:40:34 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ea6c106600
Apr 22 18:40:35 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f4c0b33e00
Apr 22 18:40:36 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1b850a400
Apr 22 18:40:36 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e42d30ee00
Apr 22 18:40:36 fir-io2-s2 kernel: LustreError: 85407:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff90f11554c450 x1631321942442272/t0(0) o3->0c4bf4e2-0960-f3e7-ed95-ab5d8a102e2a@10.8.10.6@o2ib6:384/0 lens 488/440 e 2 to 0 dl 1555983659 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:40:36 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ed17c08c00
Apr 22 18:40:36 fir-io2-s2 kernel: LustreError: 85407:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 1 previous similar message
Apr 22 18:40:36 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7362a3000
Apr 22 18:40:36 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7362a3000
Apr 22 18:40:36 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7362a3000
Apr 22 18:40:36 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7362a3000
Apr 22 18:40:36 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7362a3000
Apr 22 18:40:36 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7362a3000
Apr 22 18:40:37 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1d415b000
Apr 22 18:40:39 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff910247f0ca00
Apr 22 18:40:39 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d94c6d2c00
Apr 22 18:40:40 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8810e9200
Apr 22 18:40:41 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ed41e0b200
Apr 22 18:40:42 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7c1aa4600
Apr 22 18:40:43 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3a5a75200
Apr 22 18:40:43 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd2af97c00
Apr 22 18:40:43 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d77f5b1a00
Apr 22 18:40:43 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e8cc03ec00
Apr 22 18:40:43 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013daea00
Apr 22 18:40:46 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e44096fe00
Apr 22 18:40:47 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1d415ae00
Apr 22 18:40:47 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f086837200
Apr 22 18:40:47 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f086837200
Apr 22 18:40:47 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f086837200
Apr 22 18:40:48 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7362a3000
Apr 22 18:40:48 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7362a3000
Apr 22 18:40:48 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7362a3000
Apr 22 18:40:48 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7362a3000
Apr 22 18:40:52 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e836197000
Apr 22 18:40:52 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e836197000
Apr 22 18:40:52 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e836197000
Apr 22 18:40:53 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff910130880000
Apr 22 18:40:53 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9101eaf98e00
Apr 22 18:40:55 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8473fa600
Apr 22 18:40:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dbf44cdc00
Apr 22 18:40:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f228557600
Apr 22 18:40:59 fir-io2-s2 kernel: LustreError: 35440:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk READ after 100+0s  req@ffff90fea9f24c50 x1631300062978368/t0(0) o3->a3b6485f-4f02-6f2d-4f9c-1f19f04443c8@10.8.12.25@o2ib6:404/0 lens 488/440 e 2 to 0 dl 1555983679 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:40:59 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d4f3e8b400
Apr 22 18:41:00 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f6626fe800
Apr 22 18:41:00 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e5c367a200
Apr 22 18:41:00 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d79752ec00
Apr 22 18:41:00 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dadc00
Apr 22 18:41:01 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dbf44cc000
Apr 22 18:41:03 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e753127200
Apr 22 18:41:03 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e6b1a26800
Apr 22 18:41:04 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ece3081000
Apr 22 18:41:04 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f4e2ac7c00
Apr 22 18:41:05 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.8.29.2@o2ib6  ns: filter-fir-OST0015_UUID lock: ffff9100cd76ad00/0x4ec9aa14f857aec3 lrc: 4/0,0 mode: PW/PW res: [0x740000400:0x577115:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x60000400010020 nid: 10.8.29.2@o2ib6 remote: 0xcfc448c724a3c906 expref: 29 pid: 84767 timeout: 29837 lvb_type: 0
Apr 22 18:41:05 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f228553a00
Apr 22 18:41:06 fir-io2-s2 kernel: LustreError: 85043:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk READ after 100+0s  req@ffff90e11f227450 x1631545739530512/t0(0) o3->8776df39-692a-3df9-0874-72e441440742@10.8.18.22@o2ib6:410/0 lens 488/440 e 2 to 0 dl 1555983685 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:41:06 fir-io2-s2 kernel: LustreError: 85043:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 1 previous similar message
Apr 22 18:41:06 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ee54834e00
Apr 22 18:41:06 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1203c3800
Apr 22 18:41:07 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dbf44c8200
Apr 22 18:41:08 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d47bf33000
Apr 22 18:41:08 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e44b29fc00
Apr 22 18:41:08 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f0b85c2400
Apr 22 18:41:08 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013daf000
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff91008c1d3800
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb9e048400
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e44096fe00
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dac600
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dac600
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dac600
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dac600
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dac600
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dac600
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dac600
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dac600
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dac600
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dac600
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dac600
Apr 22 18:41:09 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dac600
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f228557c00
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fbc9f79200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62c7ec200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da819cdc00
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb7ac7c800
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 85373:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff90e1dd3a3c50 x1631545975344352/t0(0) o3->de6c0881-2c5e-8ab2-d83e-992caf69004f@10.8.2.33@o2ib6:415/0 lens 488/440 e 2 to 0 dl 1555983690 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d606ada600
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d606ada600
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f75e750200
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5c4bc2e00
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ecedbe0800
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e1833daa00
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f75e752c00
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f59c86f600
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9100506e3c00
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7903a1c00
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ed8d639c00
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f9931ec800
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd6af2be00
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd6af2b800
Apr 22 18:41:11 fir-io2-s2 kernel: LustreError: 85373:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 3 previous similar messages
Apr 22 18:41:13 fir-io2-s2 kernel: LustreError: 85287:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk READ after 100+0s  req@ffff91009d6ad050 x1631313909316624/t0(0) o3->db58df27-551e-acbf-8944-575be218e3f1@10.8.20.3@o2ib6:406/0 lens 520/440 e 2 to 0 dl 1555983681 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:41:13 fir-io2-s2 kernel: LustreError: 85287:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 3 previous similar messages
Apr 22 18:41:18 fir-io2-s2 kernel: LustreError: 85234:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 100+0s  req@ffff90f05e3f1450 x1631534459720800/t0(0) o4->7cd55859-82c0-c743-7e59-86aed70e9d98@10.8.17.14@o2ib6:409/0 lens 488/448 e 3 to 0 dl 1555983684 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:41:18 fir-io2-s2 kernel: LustreError: 85234:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 3 previous similar messages
Apr 22 18:41:25 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1d415ae00
Apr 22 18:41:25 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db48356a00
Apr 22 18:41:25 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e63b752000
Apr 22 18:41:25 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fabedaf400
Apr 22 18:41:25 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9101eaf98e00
Apr 22 18:41:25 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e669b1e000
Apr 22 18:41:33 fir-io2-s2 kernel: LustreError: 85374:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk WRITE after 100+0s  req@ffff90e2205ed850 x1631343918771680/t0(0) o4->7bd3af25-bee7-c5ba-c622-d03b941d9a42@10.8.29.6@o2ib6:424/0 lens 488/448 e 2 to 0 dl 1555983699 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:41:33 fir-io2-s2 kernel: LustreError: 85374:0:(ldlm_lib.c:3248:target_bulk_io()) Skipped 2 previous similar messages
Apr 22 18:41:37 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dbe0286c00
Apr 22 18:41:43 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f42e3f3200
Apr 22 18:41:43 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ece3081000
Apr 22 18:41:43 fir-io2-s2 kernel: LustreError: 85173:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff90fef7299850 x1631542186482960/t0(0) o3->170d6268-ca7a-a7d0-0083-35fb42e90690@10.8.7.33@o2ib6:458/0 lens 488/440 e 4 to 0 dl 1555983733 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:41:43 fir-io2-s2 kernel: LustreError: 85173:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 5 previous similar messages
Apr 22 18:41:44 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fe8b1c1e00
Apr 22 18:41:45 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb75972400
Apr 22 18:41:46 fir-io2-s2 kernel: LustreError: 132-0: fir-OST000d: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.8.30.25@o2ib6 inode [0x240012eaa:0x155c:0x0] object 0x300000402:8171944 extent [16777216-96477183], client returned csum 0 (type 4), server csum 6e8b38db (type 4)
Apr 22 18:41:47 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef50776e00
Apr 22 18:41:47 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef50776e00
Apr 22 18:41:47 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef50776e00
Apr 22 18:41:47 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef50776e00
Apr 22 18:41:47 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef50776e00
Apr 22 18:41:47 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef50776e00
Apr 22 18:41:47 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d971682600
Apr 22 18:41:48 fir-io2-s2 kernel: LustreError: 132-0: fir-OST000d: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.8.8.3@o2ib6 inode [0x240012eaa:0x11f6:0x0] object 0x300000402:8171702 extent [46137344-46202879], client returned csum 0 (type 4), server csum ab2fcb7c (type 4)
Apr 22 18:41:48 fir-io2-s2 kernel: LustreError: Skipped 1 previous similar message
Apr 22 18:41:49 fir-io2-s2 kernel: INFO: task ll_ost_io02_004:84907 blocked for more than 120 seconds.
Apr 22 18:41:49 fir-io2-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 18:41:49 fir-io2-s2 kernel: ll_ost_io02_004 D ffff91014a5eb0c0     0 84907      2 0x00000080
Apr 22 18:41:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddda80ae00
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa91b5b44>] __lock_page+0x74/0x90
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa91b6844>] __find_lock_page+0x54/0x70
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa91b74b4>] find_or_create_page+0x34/0xa0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc16e2ea7>] osd_bufs_get+0x3a7/0x800 [osd_ldiskfs]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc180ee67>] ofd_preprw+0x6b7/0x1160 [ofd]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc1264b4f>] ? __req_capsule_get+0x15f/0x740 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc12a857b>] tgt_brw_read+0x9db/0x1e50 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc1276f86>] ? null_alloc_rs+0x186/0x340 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc123f9af>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc123fb31>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:49 fir-io2-s2 kernel: INFO: task ll_ost_io02_014:85003 blocked for more than 120 seconds.
Apr 22 18:41:49 fir-io2-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 18:41:49 fir-io2-s2 kernel: ll_ost_io02_014 D ffff91021f909040     0 85003      2 0x00000080
Apr 22 18:41:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa91b5b44>] __lock_page+0x74/0x90
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa91b6844>] __find_lock_page+0x54/0x70
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa91b74b4>] find_or_create_page+0x34/0xa0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc16e2ea7>] osd_bufs_get+0x3a7/0x800 [osd_ldiskfs]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc180ee67>] ofd_preprw+0x6b7/0x1160 [ofd]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc1264b4f>] ? __req_capsule_get+0x15f/0x740 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc12a857b>] tgt_brw_read+0x9db/0x1e50 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc1276f86>] ? null_alloc_rs+0x186/0x340 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc123f9af>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc123fb31>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:49 fir-io2-s2 kernel: INFO: task ll_ost_io01_032:85127 blocked for more than 120 seconds.
Apr 22 18:41:49 fir-io2-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 18:41:49 fir-io2-s2 kernel: ll_ost_io01_032 D ffff90ff33b0b0c0     0 85127      2 0x00000080
Apr 22 18:41:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:41:49 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa91b5b44>] __lock_page+0x74/0x90
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa91b6844>] __find_lock_page+0x54/0x70
Apr 22 18:41:50 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d47bf33000
Apr 22 18:41:50 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d47bf33000
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa91b74b4>] find_or_create_page+0x34/0xa0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc16e2ea7>] osd_bufs_get+0x3a7/0x800 [osd_ldiskfs]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc180ee67>] ofd_preprw+0x6b7/0x1160 [ofd]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1264b4f>] ? __req_capsule_get+0x15f/0x740 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc12a857b>] tgt_brw_read+0x9db/0x1e50 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1276f86>] ? null_alloc_rs+0x186/0x340 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc123f9af>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc123fb31>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:50 fir-io2-s2 kernel: LustreError: 132-0: fir-OST0011: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.8.24.23@o2ib6 inode [0x240012eaa:0x17b4:0x0] object 0x680000402:8172914 extent [19415040-19480575], client returned csum 0 (type 4), server csum 2fd086df (type 4)
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:50 fir-io2-s2 kernel: INFO: task ll_ost_io01_033:85131 blocked for more than 120 seconds.
Apr 22 18:41:50 fir-io2-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 18:41:50 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb9e048400
Apr 22 18:41:50 fir-io2-s2 kernel: ll_ost_io01_033 D ffff9100ea700000     0 85131      2 0x00000080
Apr 22 18:41:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa91b5b44>] __lock_page+0x74/0x90
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa91b6844>] __find_lock_page+0x54/0x70
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa91b74b4>] find_or_create_page+0x34/0xa0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc16e2ea7>] osd_bufs_get+0x3a7/0x800 [osd_ldiskfs]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc180ee67>] ofd_preprw+0x6b7/0x1160 [ofd]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1264b4f>] ? __req_capsule_get+0x15f/0x740 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc12a857b>] tgt_brw_read+0x9db/0x1e50 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1276f86>] ? null_alloc_rs+0x186/0x340 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc123f9af>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc123fb31>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e836193600
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:50 fir-io2-s2 kernel: INFO: task ll_ost_io02_039:85148 blocked for more than 120 seconds.
Apr 22 18:41:50 fir-io2-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 18:41:50 fir-io2-s2 kernel: ll_ost_io02_039 D ffff9100d7978000     0 85148      2 0x00000080
Apr 22 18:41:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa91b5b44>] __lock_page+0x74/0x90
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa91b6844>] __find_lock_page+0x54/0x70
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa91b74b4>] find_or_create_page+0x34/0xa0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc16e2ea7>] osd_bufs_get+0x3a7/0x800 [osd_ldiskfs]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc180ee67>] ofd_preprw+0x6b7/0x1160 [ofd]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1264b4f>] ? __req_capsule_get+0x15f/0x740 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc12a857b>] tgt_brw_read+0x9db/0x1e50 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1276f6d>] ? null_alloc_rs+0x16d/0x340 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1276f86>] ? null_alloc_rs+0x186/0x340 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc123f9af>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc123fb31>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:50 fir-io2-s2 kernel: INFO: task ll_ost_io03_022:85191 blocked for more than 120 seconds.
Apr 22 18:41:50 fir-io2-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 18:41:50 fir-io2-s2 kernel: ll_ost_io03_022 D ffff90feafb29040     0 85191      2 0x00000080
Apr 22 18:41:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:50 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f228554800
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa91b5b44>] __lock_page+0x74/0x90
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa91b6844>] __find_lock_page+0x54/0x70
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa91b74b4>] find_or_create_page+0x34/0xa0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc16e2ea7>] osd_bufs_get+0x3a7/0x800 [osd_ldiskfs]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc180ee67>] ofd_preprw+0x6b7/0x1160 [ofd]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1264b4f>] ? __req_capsule_get+0x15f/0x740 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc12a857b>] tgt_brw_read+0x9db/0x1e50 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1276f86>] ? null_alloc_rs+0x186/0x340 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc123f9af>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc123fb31>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:41:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:50 fir-io2-s2 kernel: INFO: task ll_ost_io01_054:85205 blocked for more than 120 seconds.
Apr 22 18:41:50 fir-io2-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 18:41:51 fir-io2-s2 kernel: ll_ost_io01_054 D ffff90febaa09040     0 85205      2 0x00000080
Apr 22 18:41:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa91b5b44>] __lock_page+0x74/0x90
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa91b6844>] __find_lock_page+0x54/0x70
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa91b74b4>] find_or_create_page+0x34/0xa0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc16e2ea7>] osd_bufs_get+0x3a7/0x800 [osd_ldiskfs]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc180ee67>] ofd_preprw+0x6b7/0x1160 [ofd]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1264b4f>] ? __req_capsule_get+0x15f/0x740 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9394d0d>] ? list_del+0xd/0x30
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc12a857b>] tgt_brw_read+0x9db/0x1e50 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1276f86>] ? null_alloc_rs+0x186/0x340 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc123f9af>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc123fb31>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:51 fir-io2-s2 kernel: INFO: task ll_ost_io00_074:85328 blocked for more than 120 seconds.
Apr 22 18:41:51 fir-io2-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 18:41:51 fir-io2-s2 kernel: ll_ost_io00_074 D ffff90fc7cf7e180     0 85328      2 0x00000080
Apr 22 18:41:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa91b5b44>] __lock_page+0x74/0x90
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa91b6844>] __find_lock_page+0x54/0x70
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa91b74b4>] find_or_create_page+0x34/0xa0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc16e2ea7>] osd_bufs_get+0x3a7/0x800 [osd_ldiskfs]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc180ee67>] ofd_preprw+0x6b7/0x1160 [ofd]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1264b4f>] ? __req_capsule_get+0x15f/0x740 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc12a857b>] tgt_brw_read+0x9db/0x1e50 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1276f86>] ? null_alloc_rs+0x186/0x340 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc123f9af>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc123fb31>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:51 fir-io2-s2 kernel: INFO: task ll_ost_io00_086:85376 blocked for more than 120 seconds.
Apr 22 18:41:51 fir-io2-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 18:41:51 fir-io2-s2 kernel: ll_ost_io00_086 D ffff90e2b6cd1040     0 85376      2 0x00000080
Apr 22 18:41:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa91b5b44>] __lock_page+0x74/0x90
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa91b6844>] __find_lock_page+0x54/0x70
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa91b74b4>] find_or_create_page+0x34/0xa0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc16e2ea7>] osd_bufs_get+0x3a7/0x800 [osd_ldiskfs]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc180ee67>] ofd_preprw+0x6b7/0x1160 [ofd]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1264b4f>] ? __req_capsule_get+0x15f/0x740 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc12a857b>] tgt_brw_read+0x9db/0x1e50 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1276f86>] ? null_alloc_rs+0x186/0x340 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc123f9af>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc123fb31>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90d0884>] ? finish_task_switch+0x54/0x1c0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:51 fir-io2-s2 kernel: INFO: task ll_ost_io01_092:85449 blocked for more than 120 seconds.
Apr 22 18:41:51 fir-io2-s2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 22 18:41:51 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f013c2d000
Apr 22 18:41:51 fir-io2-s2 kernel: ll_ost_io01_092 D ffff90e2b5e3c100     0 85449      2 0x00000080
Apr 22 18:41:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa91b5b44>] __lock_page+0x74/0x90
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa91b6844>] __find_lock_page+0x54/0x70
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa91b74b4>] find_or_create_page+0x34/0xa0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc16e2ea7>] osd_bufs_get+0x3a7/0x800 [osd_ldiskfs]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc180ee67>] ofd_preprw+0x6b7/0x1160 [ofd]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1264b4f>] ? __req_capsule_get+0x15f/0x740 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc12a857b>] tgt_brw_read+0x9db/0x1e50 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1276f86>] ? null_alloc_rs+0x186/0x340 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc123f9af>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc123fb31>] ? lustre_pack_reply+0x11/0x20 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:41:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:41:53 fir-io2-s2 kernel: LustreError: 88362:0:(tgt_handler.c:644:process_req_last_xid()) @@@ Unexpected xid 5cbb2ac7dab80 vs. last_xid 5cbb2ac7de2bf#012  req@ffff90e6ac0e0850 x1631343142087552/t0(0) o4->43698ef5-a64b-9a4b-3198-ccc30362bbed@10.8.23.35@o2ib6:445/0 lens 1336/0 e 0 to 0 dl 1555983720 ref 1 fl Interpret:/2/ffffffff rc 0/-1
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e9e8128c00
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d4f3e89400
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f6b9d34000
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1f3701e00
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e954928600
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e013dade00
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db1acec000
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db1acec000
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5c4bc3c00
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e95492ea00
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e95492ea00
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e95492ea00
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e95492ea00
Apr 22 18:41:56 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da819ca400
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e67a7d9600
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dbe0288400
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7903a1c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dffb9dc000
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f62a293a00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3e2639e00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f9f4658600
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dc65600200
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dfc2098a00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f89da7e200
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9100f84be400
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d929b75c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d929b75c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d929b75c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9101a37ab400
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd8efd7600
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f50dbb4600
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1a5b95400
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ecc4297e00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90feeaff6600
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e754eec400
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f9931ec800
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec9eae5c00
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e277c08800
Apr 22 18:41:57 fir-io2-s2 kernel: LustreError: 132-0: fir-OST0013: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.8.13.7@o2ib6 inode [0x240012eaa:0x180e:0x0] object 0x600000402:8169842 extent [211050496-211116031], client returned csum 0 (type 4), server csum 807c14b7 (type 4)
Apr 22 18:41:58 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d88116a200
Apr 22 18:41:58 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dbf44ce000
Apr 22 18:42:04 fir-io2-s2 kernel: LustreError: 85215:0:(ldlm_lib.c:3248:target_bulk_io()) @@@ timeout on bulk READ after 100+0s  req@ffff90f2a8e84c50 x1631545753225936/t0(0) o3->0c79f29c-4464-0e4c-f90d-52cb866bcabd@10.8.18.11@o2ib6:461/0 lens 488/440 e 4 to 0 dl 1555983736 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:42:08 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d47bf31c00
Apr 22 18:42:12 fir-io2-s2 kernel: Lustre: fir-OST000d: Connection restored to 3c3bbb00-493c-e0ea-5625-2178da530699 (at 10.8.2.32@o2ib6)
Apr 22 18:42:12 fir-io2-s2 kernel: Lustre: Skipped 5053 previous similar messages
Apr 22 18:42:14 fir-io2-s2 kernel: LNetError: 81272:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 18:42:14 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db1acec000
Apr 22 18:42:14 fir-io2-s2 kernel: Lustre: fir-OST0015: Bulk IO read error with 98e3664a-d5ef-eae6-fd14-b2cecf487ed2 (at 10.8.12.10@o2ib6), client will retry: rc -110
Apr 22 18:42:14 fir-io2-s2 kernel: Lustre: Skipped 173 previous similar messages
Apr 22 18:42:14 fir-io2-s2 kernel: LNetError: 81272:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 255 previous similar messages
Apr 22 18:42:14 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb75976e00
Apr 22 18:42:14 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90edd85ec400
Apr 22 18:42:14 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9100ea76f800
Apr 22 18:42:14 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eac6431800
Apr 22 18:42:14 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e97ef6b800
Apr 22 18:42:14 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9100ea76fe00
Apr 22 18:42:14 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e389e51c00
Apr 22 18:42:14 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f5e0e43000
Apr 22 18:42:14 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f5e0e43000
Apr 22 18:42:14 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f5e0e43000
Apr 22 18:42:15 fir-io2-s2 kernel: Lustre: fir-OST0011: Bulk IO write error with 301bd1d5-f294-00c7-57bb-0517ce6cb157 (at 10.8.15.8@o2ib6), client will retry: rc = -110
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de0f39ea00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e9bbc9fc00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d65655de00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f98cb6da00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7c2e79600
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7a79c4e00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7c2e79800
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e667f0cc00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f9931ec800
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eadb3eac00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fb8a6ab600
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3efdb5600
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f28814d400
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f86b022c00
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f28814b400
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90daf3fa5600
Apr 22 18:42:15 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fb1a7fc800
Apr 22 18:42:15 fir-io2-s2 kernel: Lustre: 83743:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (118:45s); client may timeout.  req@ffff90ef4dda1c50 x1631388870162480/t0(0) o101->63d5b964-3268-0813-20ce-ce2c87bbd1d8@10.8.29.2@o2ib6:414/0 lens 328/368 e 3 to 0 dl 1555983689 ref 1 fl Complete:/0/0 rc 301/301
Apr 22 18:42:15 fir-io2-s2 kernel: Lustre: Skipped 21 previous similar messages
Apr 22 18:43:37 fir-io2-s2 kernel: Lustre: fir-OST000f: Client 0c79f29c-4464-0e4c-f90d-52cb866bcabd (at 10.8.18.11@o2ib6) reconnecting
Apr 22 18:43:37 fir-io2-s2 kernel: Lustre: Skipped 3216 previous similar messages
Apr 22 18:44:20 fir-io2-s2 kernel: Lustre: 84795:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555983853/real 0]  req@ffff90f761947500 x1631537036502688/t0(0) o104->fir-OST0015@10.8.2.18@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555983860 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Apr 22 18:44:20 fir-io2-s2 kernel: Lustre: 84795:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 137 previous similar messages
Apr 22 18:44:52 fir-io2-s2 kernel: LustreError: 85457:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff90f2bbfa5450 x1631534473434640/t0(0) o4->38fc721f-2581-5cc7-2331-7b71af28244a@10.8.7.30@o2ib6:16/0 lens 488/448 e 1 to 0 dl 1555984046 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:44:52 fir-io2-s2 kernel: LustreError: 85457:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 130 previous similar messages
Apr 22 18:45:08 fir-io2-s2 kernel: LustreError: 85137:0:(ldlm_lib.c:3273:target_bulk_io()) @@@ truncated bulk READ 0(65536)  req@ffff90f10f217850 x1631313909509312/t0(0) o3->db58df27-551e-acbf-8944-575be218e3f1@10.8.20.3@o2ib6:18/0 lens 488/440 e 1 to 0 dl 1555984048 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:45:12 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef118cf400
Apr 22 18:45:12 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef118cf400
Apr 22 18:45:12 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef118cf400
Apr 22 18:45:12 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef118cf400
Apr 22 18:45:14 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef118cf400
Apr 22 18:45:14 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef118cf400
Apr 22 18:45:14 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef118cf400
Apr 22 18:45:14 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef118cf400
Apr 22 18:45:37 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fac6acdc00
Apr 22 18:45:37 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fac6acdc00
Apr 22 18:45:51 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185aa00
Apr 22 18:45:51 fir-io2-s2 kernel: LustreError: 82410:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff90fd0f167c50 x1628555668924608/t0(0) o4->a9a3cba7-797a-18c8-deb5-3950fcb66f75@10.8.7.12@o2ib6:95/0 lens 488/448 e 0 to 0 dl 1555984125 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:45:52 fir-io2-s2 kernel: LustreError: 82410:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 25 previous similar messages
Apr 22 18:45:52 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fac6acdc00
Apr 22 18:45:54 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fac6acdc00
Apr 22 18:45:58 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e1a9b00200
Apr 22 18:46:01 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:01 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:01 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:01 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:02 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e992249c00
Apr 22 18:46:06 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e17d986e00
Apr 22 18:46:07 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5c1d8c200
Apr 22 18:46:07 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa4515f000
Apr 22 18:46:07 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddf16bee00
Apr 22 18:46:07 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddf16bee00
Apr 22 18:46:07 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddf16bee00
Apr 22 18:46:07 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddf16bee00
Apr 22 18:46:07 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddf16bee00
Apr 22 18:46:08 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efbac17200
Apr 22 18:46:08 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5c1d8c200
Apr 22 18:46:08 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa4515f000
Apr 22 18:46:09 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e57c6c2a00
Apr 22 18:46:12 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:12 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:12 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:12 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:12 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:12 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:12 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:12 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:12 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:12 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:12 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:12 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f49185e800
Apr 22 18:46:16 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efbac17200
Apr 22 18:46:16 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5c1d8c200
Apr 22 18:46:16 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa4515f000
Apr 22 18:46:16 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddf16bee00
Apr 22 18:46:16 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddf16bee00
Apr 22 18:46:16 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddf16bee00
Apr 22 18:46:17 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddf16bee00
Apr 22 18:46:17 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddf16bee00
Apr 22 18:46:17 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddf16bee00
Apr 22 18:46:17 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddf16bee00
Apr 22 18:46:17 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efbac17200
Apr 22 18:46:17 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efbac17200
Apr 22 18:46:17 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5c1d8c200
Apr 22 18:46:17 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fa4515f000
Apr 22 18:46:17 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e93bf9fa00
Apr 22 18:46:44 fir-io2-s2 kernel: LustreError: 137-5: fir-OST000c_UUID: not available for connect from 10.8.12.13@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server.
Apr 22 18:46:44 fir-io2-s2 kernel: LustreError: Skipped 370 previous similar messages
Apr 22 18:47:53 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 10.9.107.60@o2ib4  ns: filter-fir-OST0015_UUID lock: ffff90f3de829d40/0x4ec9aa14f8ccf331 lrc: 3/0,0 mode: PW/PW res: [0x740000401:0x3f5e03:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000000020020 nid: 10.9.107.60@o2ib4 remote: 0x431878887b9ef861 expref: 76 pid: 84795 timeout: 30245 lvb_type: 0
Apr 22 18:48:24 fir-io2-s2 kernel: LNetError: 81272:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)
Apr 22 18:48:24 fir-io2-s2 kernel: LNetError: 81272:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 104 previous similar messages
Apr 22 18:48:24 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da2382da00
Apr 22 18:48:27 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da2382da00
Apr 22 18:48:27 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da2382da00
Apr 22 18:48:28 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da2382da00
Apr 22 18:48:28 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da2382da00
Apr 22 18:48:28 fir-io2-s2 kernel: Lustre: fir-OST0011: Bulk IO write error with 38fc721f-2581-5cc7-2331-7b71af28244a (at 10.8.7.30@o2ib6), client will retry: rc = -110
Apr 22 18:48:28 fir-io2-s2 kernel: Lustre: Skipped 7 previous similar messages
Apr 22 18:48:53 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e22fd35000
Apr 22 18:48:53 fir-io2-s2 kernel: LustreError: 38519:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff90e109b76850 x1631329926812992/t0(0) o4->95e6fd6a-706d-ff18-fa02-0b0e9d53d014@10.8.19.8@o2ib6:301/0 lens 488/448 e 0 to 0 dl 1555984331 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:48:53 fir-io2-s2 kernel: LustreError: 38519:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 10 previous similar messages
Apr 22 18:48:53 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7aeb1bc00
Apr 22 18:48:54 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e609222800
Apr 22 18:48:56 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df4819f800
Apr 22 18:48:56 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df4819f800
Apr 22 18:48:56 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df4819f800
Apr 22 18:48:56 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df4819f800
Apr 22 18:48:56 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df4819f800
Apr 22 18:48:56 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df4819f800
Apr 22 18:48:56 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df4819f800
Apr 22 18:48:56 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90df4819f800
Apr 22 18:48:58 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3e2026600
Apr 22 18:48:58 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3e2026600
Apr 22 18:48:58 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3e2026600
Apr 22 18:48:58 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3e2026600
Apr 22 18:49:02 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebc8b09a00
Apr 22 18:49:04 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f11113b400
Apr 22 18:49:04 fir-io2-s2 kernel: Lustre: fir-OST0011: Bulk IO read error with 176b6ecb-736c-d192-973a-179bdc2a9222 (at 10.8.26.27@o2ib6), client will retry: rc -110
Apr 22 18:49:04 fir-io2-s2 kernel: Lustre: Skipped 48 previous similar messages
Apr 22 18:49:04 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f6532af600
Apr 22 18:49:05 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3e2026600
Apr 22 18:49:05 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3e2026600
Apr 22 18:49:05 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3e2026600
Apr 22 18:49:05 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d90cdf7000
Apr 22 18:49:05 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddcbacb000
Apr 22 18:49:06 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dc4fbc4c00
Apr 22 18:49:07 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90da215a1e00
Apr 22 18:49:07 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e85b5cca00
Apr 22 18:49:07 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebc8b09a00
Apr 22 18:49:07 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebc8b09a00
Apr 22 18:49:07 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebc8b09a00
Apr 22 18:49:08 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0be650c00
Apr 22 18:49:09 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dc4fbc2c00
Apr 22 18:49:09 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3fa366c00
Apr 22 18:49:09 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9100bf712600
Apr 22 18:49:09 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd513c7a00
Apr 22 18:49:09 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d90cdfb800
Apr 22 18:49:09 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddcbacb000
Apr 22 18:49:09 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d43b81c600
Apr 22 18:49:10 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebd2455600
Apr 22 18:49:10 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebd2455600
Apr 22 18:49:10 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebd2455600
Apr 22 18:49:10 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d6619e9e00
Apr 22 18:49:10 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebd2455600
Apr 22 18:49:11 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e71c13a800
Apr 22 18:49:11 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e71c13a800
Apr 22 18:49:11 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e71c13a800
Apr 22 18:49:11 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f491782800
Apr 22 18:49:11 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f491786600
Apr 22 18:49:11 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de7a709e00
Apr 22 18:49:11 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd513c7a00
Apr 22 18:49:12 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dd3720f400
Apr 22 18:49:12 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7a9ffac00
Apr 22 18:49:12 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7a9ffac00
Apr 22 18:49:12 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7a9ffac00
Apr 22 18:49:12 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7a9ffac00
Apr 22 18:49:12 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec59af1200
Apr 22 18:49:12 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec59af1200
Apr 22 18:49:12 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec59af1200
Apr 22 18:49:12 fir-io2-s2 kernel: LustreError: 85095:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff90feb95ac850 x1631545920106320/t0(0) o3->1b4aa412-a51a-3403-4e66-b5357add918b@10.8.2.25@o2ib6:323/0 lens 488/440 e 0 to 0 dl 1555984353 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:49:12 fir-io2-s2 kernel: LustreError: 85095:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 6 previous similar messages
Apr 22 18:49:12 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f97ccca000
Apr 22 18:49:12 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de7a708c00
Apr 22 18:49:13 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d43b81c600
Apr 22 18:49:14 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fe09db5400
Apr 22 18:49:14 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e02f93fc00
Apr 22 18:49:16 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90edd3d91600
Apr 22 18:49:16 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fe09db5400
Apr 22 18:49:16 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fe09db5400
Apr 22 18:49:16 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fe09db5400
Apr 22 18:49:17 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f520309e00
Apr 22 18:49:18 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dcb85f9200
Apr 22 18:49:20 fir-io2-s2 kernel: Lustre: 109373:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555984153/real 0]  req@ffff90fecc481800 x1631537039638416/t0(0) o105->fir-OST0011@10.8.7.35@o2ib6:15/16 lens 360/224 e 0 to 1 dl 1555984160 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Apr 22 18:49:20 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e8ceaf2a00
Apr 22 18:49:20 fir-io2-s2 kernel: Lustre: 109373:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 369 previous similar messages
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff910123ea6000
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d868acd200
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f39a5b1c00
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f39a5b1c00
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f39a5b1c00
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f39a5b1c00
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f39a5b1c00
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fbcb6ae000
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90daae938600
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e48a72f400
Apr 22 18:49:21 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d2be26be00
Apr 22 18:49:22 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ee11f1a200
Apr 22 18:49:22 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1f847b200
Apr 22 18:49:22 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb4236ae00
Apr 22 18:49:22 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f7ccf29e00
Apr 22 18:49:23 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e31d07ba00
Apr 22 18:49:23 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e31d07ba00
Apr 22 18:49:23 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e31d07ba00
Apr 22 18:49:23 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f537d40600
Apr 22 18:49:23 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f537d40600
Apr 22 18:49:23 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eae2943800
Apr 22 18:49:24 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d620975a00
Apr 22 18:49:24 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d620975a00
Apr 22 18:49:24 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d620975a00
Apr 22 18:49:24 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d620975a00
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ec3fe79a00
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff910123ea4a00
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d868acd200
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f39a5b1c00
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f39a5b1c00
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f39a5b1c00
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f39a5b1c00
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f39a5b1c00
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f39a5b1c00
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f39a5b1c00
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e28bc6c600
Apr 22 18:49:25 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fbcb6ae000
Apr 22 18:49:26 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dee2961800
Apr 22 18:49:26 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e48a72f400
Apr 22 18:49:26 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dc4fbc0000
Apr 22 18:49:26 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d769014600
Apr 22 18:49:26 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9101f7dd7400
Apr 22 18:49:26 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb4236ae00
Apr 22 18:49:26 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dbc4445400
Apr 22 18:49:26 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e1fcf02600
Apr 22 18:49:27 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e31d07ba00
Apr 22 18:49:27 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f537d40600
Apr 22 18:49:27 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f537d40600
Apr 22 18:49:27 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eae2943800
Apr 22 18:49:27 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db56744000
Apr 22 18:49:27 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e1fcf02600
Apr 22 18:49:27 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff63120c00
Apr 22 18:49:27 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f35799d200
Apr 22 18:49:27 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fc77743e00
Apr 22 18:49:27 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fc77743e00
Apr 22 18:49:27 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fc77743e00
Apr 22 18:49:27 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fc77743e00
Apr 22 18:49:28 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8b2a4e200
Apr 22 18:49:28 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8b2a4e200
Apr 22 18:49:28 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8b2a4e200
Apr 22 18:49:28 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f895ec6e00
Apr 22 18:49:28 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f895ec6e00
Apr 22 18:49:28 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d4bc056000
Apr 22 18:49:28 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d4bc056000
Apr 22 18:49:28 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d4bc056000
Apr 22 18:49:29 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f52efda800
Apr 22 18:49:29 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d6e56fbe00
Apr 22 18:49:29 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d7b59a1600
Apr 22 18:49:29 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f52efda800
Apr 22 18:49:30 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e987b95800
Apr 22 18:49:31 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e987b95600
Apr 22 18:49:32 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dc6e89e400
Apr 22 18:49:32 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f513709800
Apr 22 18:49:32 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f45fecd200
Apr 22 18:49:32 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d3d8857a00
Apr 22 18:49:33 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb8e880400
Apr 22 18:49:33 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb8e880400
Apr 22 18:49:33 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb8e880400
Apr 22 18:49:33 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb8e880400
Apr 22 18:49:33 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb8e880400
Apr 22 18:49:33 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb8e880400
Apr 22 18:49:34 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d51c1c9e00
Apr 22 18:49:35 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dc6e89b400
Apr 22 18:49:35 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f513709800
Apr 22 18:49:35 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e6b5500c00
Apr 22 18:49:35 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e77def0a00
Apr 22 18:49:35 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efe5b5de00
Apr 22 18:49:35 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb8e88d000
Apr 22 18:49:35 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9100ff9b1000
Apr 22 18:49:36 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5d59f3200
Apr 22 18:49:36 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d51c1c9e00
Apr 22 18:49:36 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d51c1c9e00
Apr 22 18:49:37 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f0ab13c400
Apr 22 18:49:39 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f51c680c00
Apr 22 18:49:40 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ee395cc600
Apr 22 18:49:41 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fe76f59c00
Apr 22 18:49:41 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ef92d5b600
Apr 22 18:49:43 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ebfd8c0200
Apr 22 18:49:43 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fbecb24600
Apr 22 18:49:43 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff910153808c00
Apr 22 18:49:43 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5f19fcc00
Apr 22 18:49:43 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f15c536800
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5f4a65a00
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e8ceaf2a00
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f334de6600
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d57a9df800
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5cff91e00
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e88c942e00
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d8f1964000
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d756705c00
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f70fc59800
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f70fc5de00
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f9001c0400
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db5bf8f800
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90edec4d4e00
Apr 22 18:49:44 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddb5c3ce00
Apr 22 18:49:45 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e4bf4bec00
Apr 22 18:49:45 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff91018913f000
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f170441e00
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f170441e00
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f170441e00
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f170441e00
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f170441e00
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb2a17de00
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb2a17de00
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb2a17de00
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb2a17de00
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb2a17de00
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e662838600
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ed05638400
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fe76f45c00
Apr 22 18:49:46 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3b2991a00
Apr 22 18:49:47 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f69c4ed600
Apr 22 18:49:47 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eaa330ee00
Apr 22 18:49:47 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e5f13ae600
Apr 22 18:49:47 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e41fd29400
Apr 22 18:49:48 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff9eed7800
Apr 22 18:49:48 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd6af18800
Apr 22 18:49:48 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d72696b800
Apr 22 18:49:48 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e6a1200600
Apr 22 18:49:48 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e6a1200600
Apr 22 18:49:48 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e6a1200600
Apr 22 18:49:48 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d3c2095400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1fad1f400
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5f19fcc00
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fdc3936c00
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d657832600
Apr 22 18:49:49 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f077b03000
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ea9d089a00
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90de61004000
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fb9c24c400
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d93af5ac00
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fc3cf93e00
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dc07977600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f1aa542200
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e3b15e0600
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e642b9be00
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff91020cd29c00
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e6a6e00e00
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff9eed5000
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81271:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e22f663e00
Apr 22 18:49:50 fir-io2-s2 kernel: LustreError: 81268:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e22f663e00
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81269:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ddeb114200
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e059336e00
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e4bf4bec00
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e5b425d600
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f627da5e00
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f170441e00
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f170441e00
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f170441e00
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f170441e00
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81276:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5b2b01800
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90deb5164800
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e88c7d8200
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5cff91e00
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5cff91e00
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e62b444000
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e62b444000
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e62b444000
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f07dba5200
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f07dba5200
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f07dba5200
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f07dba5200
Apr 22 18:49:51 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8c874a400
Apr 22 18:49:52 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f3d101ea00
Apr 22 18:49:52 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db5bf8f800
Apr 22 18:49:52 fir-io2-s2 kernel: LustreError: 81277:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd6af18800
Apr 22 18:49:52 fir-io2-s2 kernel: LustreError: 81279:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d72696b800
Apr 22 18:49:52 fir-io2-s2 kernel: LustreError: 81278:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0543f2c00
Apr 22 18:49:54 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e6a1200600
Apr 22 18:49:54 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e6a1200600
Apr 22 18:49:54 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e6a1200600
Apr 22 18:49:55 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f69c4ee800
Apr 22 18:49:55 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db10aac400
Apr 22 18:49:56 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e4bf4bec00
Apr 22 18:49:56 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dcd8826c00
Apr 22 18:49:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb2a17de00
Apr 22 18:49:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb2a17de00
Apr 22 18:49:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb2a17de00
Apr 22 18:49:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb2a17de00
Apr 22 18:49:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb2a17de00
Apr 22 18:49:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb2a17de00
Apr 22 18:49:57 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90eb2a17de00
Apr 22 18:49:57 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff9eed7200
Apr 22 18:49:58 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fae791d600
Apr 22 18:49:58 fir-io2-s2 kernel: LustreError: 81270:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff91022378b000
Apr 22 18:49:58 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7a4261a00
Apr 22 18:49:58 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f0b1828000
Apr 22 18:49:58 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90fd6af18800
Apr 22 18:49:59 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d84116d000
Apr 22 18:49:59 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e0543f2c00
Apr 22 18:50:00 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d5f19fcc00
Apr 22 18:50:08 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ee72856800
Apr 22 18:50:10 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e10ec91600
Apr 22 18:50:20 fir-io2-s2 kernel: bash (46045): drop_caches: 1
Apr 22 18:50:33 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f8f51c3200
Apr 22 18:50:33 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f0616cc800
Apr 22 18:50:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555dc00
Apr 22 18:50:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555dc00
Apr 22 18:50:33 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dbc9cca000
Apr 22 18:50:33 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dbc9cca000
Apr 22 18:50:33 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dbc9cca000
Apr 22 18:50:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90dbc9cca000
Apr 22 18:50:33 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7dff6a400
Apr 22 18:50:33 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7dff6a400
Apr 22 18:50:33 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7dff6a400
Apr 22 18:50:33 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7dff6a400
Apr 22 18:50:34 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7dff6a400
Apr 22 18:50:34 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7dff6a400
Apr 22 18:50:34 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e7dff6a400
Apr 22 18:50:34 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d9738c3800
Apr 22 18:50:34 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d9738c3800
Apr 22 18:50:34 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d9738c3800
Apr 22 18:50:34 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d9738c3800
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555f600
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555ee00
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555ee00
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555ee00
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555ee00
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555ee00
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555ee00
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555ee00
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555ee00
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555ee00
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555ee00
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555ee00
Apr 22 18:50:37 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90e16555ee00
Apr 22 18:50:38 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90efe1947200
Apr 22 18:50:38 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db815cca00
Apr 22 18:50:38 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db815cca00
Apr 22 18:50:38 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db815cca00
Apr 22 18:50:38 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db815cca00
Apr 22 18:50:38 fir-io2-s2 kernel: LustreError: 81275:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90db815cca00
Apr 22 18:50:39 fir-io2-s2 kernel: LustreError: 81274:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90ff89220600
Apr 22 18:50:39 fir-io2-s2 kernel: LustreError: 81273:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90d92f13dc00
Apr 22 18:50:39 fir-io2-s2 kernel: LustreError: 81272:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff90f956cbc200
Apr 22 18:50:41 fir-io2-s2 kernel: LustreError: 85268:0:(ldlm_lib.c:3207:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff90e0937cf050 x1631477069409360/t0(0) o3->99fcecbc-de72-33fa-af25-544df34bc1b4@10.8.26.3@o2ib6:225/0 lens 488/440 e 2 to 0 dl 1555984255 ref 1 fl Interpret:/0/0 rc 0/0
Apr 22 18:50:41 fir-io2-s2 kernel: LustreError: 85268:0:(ldlm_lib.c:3207:target_bulk_io()) Skipped 41 previous similar messages
Apr 22 18:51:16 fir-io2-s2 kernel: SysRq : Show State
Apr 22 18:51:16 fir-io2-s2 kernel:  task                        PC stack   pid father
Apr 22 18:51:16 fir-io2-s2 kernel: systemd         S ffff90e369e88000     0     1      0 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9766e7d>] schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa928eaa9>] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa928ed3e>] ep_poll+0x23e/0x360
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa929020d>] SyS_epoll_wait+0xed/0x120
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:16 fir-io2-s2 kernel: kthreadd        S ffff90e369e89040     0     2      0 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c2625>] kthreadd+0x2f5/0x300
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c2330>] ? kthread_create_on_cpu+0x60/0x60
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c2330>] ? kthread_create_on_cpu+0x60/0x60
Apr 22 18:51:16 fir-io2-s2 kernel: ksoftirqd/0     S ffff90d3e9b60000     0     3      2 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: kworker/0:0H    S ffff90d3e9b62080     0     5      2 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: migration/0     S ffff90d3e9b65140     0     8      2 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: rcu_bh          S ffff90d3e9b66180     0     9      2 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa91554c2>] rcu_gp_kthread+0xb2/0x710
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9155410>] ? rcu_gp_fqs+0x90/0x90
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: rcu_sched       S ffff90d3e94a8000     0    10      2 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c27f6>] ? prepare_to_wait+0x56/0x90
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9155759>] rcu_gp_kthread+0x349/0x710
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9155410>] ? rcu_gp_fqs+0x90/0x90
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: lru-add-drain   S ffff90d3e94a9040     0    11      2 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: watchdog/0      S ffff90d3e94aa080     0    12      2 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9148c40>] ? touch_nmi_watchdog+0x30/0x30
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: watchdog/1      S ffff90e369fc30c0     0    13      2 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: migration/1     S ffff90e369ff8000     0    14      2 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa912ff20>] ? cpu_stop_queue_work+0x70/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: ksoftirqd/1     S ffff90e369ff9040     0    15      2 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:16 fir-io2-s2 kernel: kworker/1:0H    S ffff90e369ffb0c0     0    17      2 0x00000000
Apr 22 18:51:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:16 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: watchdog/2      S ffff90f369f7a080     0    19      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: migration/2     S ffff90f369f7b0c0     0    20      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: ksoftirqd/2     S ffff90f369f7c100     0    21      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: kworker/2:0H    S ffff90f369f99040     0    23      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: watchdog/3      S ffff910369452080     0    25      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: migration/3     S ffff9103694530c0     0    26      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: ksoftirqd/3     S ffff910369454100     0    27      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: kworker/3:0H    S ffff910369481040     0    29      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: watchdog/4      S ffff90d3e953b0c0     0    31      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9148c40>] ? touch_nmi_watchdog+0x30/0x30
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: migration/4     S ffff90d3e9578000     0    32      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: ksoftirqd/4     S ffff90d3e9579040     0    33      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: kworker/4:0H    S ffff90d3e957b0c0     0    35      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: watchdog/5      S ffff90e369ffe180     0    36      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: migration/5     S ffff90e369ffd140     0    37      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: ksoftirqd/5     S ffff90e369838000     0    38      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: kworker/5:0H    S ffff90e36983a080     0    40      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: watchdog/6      S ffff90f369f9e180     0    41      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9148c40>] ? touch_nmi_watchdog+0x30/0x30
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: migration/6     S ffff90f369f9d140     0    42      2 0x00000000
Apr 22 18:51:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: ksoftirqd/6     S ffff90f369f9c100     0    43      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: kworker/6:0H    S ffff90f369fd8000     0    45      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: watchdog/7      S ffff910369486180     0    46      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9148c40>] ? touch_nmi_watchdog+0x30/0x30
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: migration/7     S ffff910369485140     0    47      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: ksoftirqd/7     S ffff910369484100     0    48      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: kworker/7:0H    S ffff9103694e8000     0    50      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: watchdog/8      S ffff90d3e957e180     0    51      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9148c40>] ? touch_nmi_watchdog+0x30/0x30
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: migration/8     S ffff90d3e957d140     0    52      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: ksoftirqd/8     S ffff90d3e957c100     0    53      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: kworker/8:0H    S ffff90d3e95e1040     0    55      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: watchdog/9      S ffff90e36983e180     0    56      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: migration/9     S ffff90e36983d140     0    57      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: ksoftirqd/9     S ffff90e36983c100     0    58      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: kworker/9:0H    S ffff90f2bbc9a580     0    60      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: watchdog/10     S ffff90f369fde180     0    61      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: migration/10    S ffff90f369fdd140     0    62      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: ksoftirqd/10    S ffff90f369fdc100     0    63      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: kworker/10:0H   S ffff90f369fda080     0    65      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: watchdog/11     S ffff9103694ee180     0    66      2 0x00000000
Apr 22 18:51:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: migration/11    S ffff9103694ed140     0    67      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: ksoftirqd/11    S ffff9103694ec100     0    68      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: kworker/11:0H   S ffff9103694ea080     0    70      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: watchdog/12     R  running task        0    71      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: migration/12    R  running task        0    72      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa912ff20>] ? cpu_stop_queue_work+0x70/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: ksoftirqd/12    R  running task        0    73      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: kworker/12:0    R  running task        0    74      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: kworker/12:0H   S ffff90d3e95e2080     0    75      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: watchdog/13     S ffff90e369876180     0    76      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: migration/13    S ffff90e369875140     0    77      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: ksoftirqd/13    S ffff90e369874100     0    78      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: kworker/13:0H   S ffff90e369872080     0    80      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: watchdog/14     S ffff90f369fd9040     0    81      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9148c40>] ? touch_nmi_watchdog+0x30/0x30
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: migration/14    S ffff90f369438000     0    82      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa912ff20>] ? cpu_stop_queue_work+0x70/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: ksoftirqd/14    S ffff90f369439040     0    83      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: kworker/14:0    S ffff90f36943a080     0    84      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: kworker/14:0H   S ffff90f36943b0c0     0    85      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: watchdog/15     S ffff9103694e9040     0    86      2 0x00000000
Apr 22 18:51:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: migration/15    S ffff910369530000     0    87      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: ksoftirqd/15    S ffff910369531040     0    88      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: kworker/15:0H   S ffff9103695330c0     0    90      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: watchdog/16     S ffff90d3e9650000     0    91      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9148df0>] ? watchdog_cleanup+0x10/0x10
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: migration/16    S ffff90d3e9651040     0    92      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa976a4da>] ? _raw_spin_lock_irqsave+0xa/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: ksoftirqd/16    S ffff90d3e9652080     0    93      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: kworker/16:0H   S ffff90d3e9654100     0    95      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: watchdog/17     S ffff90e369871040     0    96      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: migration/17    S ffff90e3698c0000     0    97      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: ksoftirqd/17    S ffff90e3698c1040     0    98      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: kworker/17:0    S ffff90f2bb449f40     0    99      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: kworker/17:0H   S ffff90e3698c30c0     0   100      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: watchdog/18     S ffff90f36943e180     0   101      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9148df0>] ? watchdog_cleanup+0x10/0x10
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: migration/18    S ffff90f36943d140     0   102      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: ksoftirqd/18    S ffff90f36943c100     0   103      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: kworker/18:0H   S ffff90f369471040     0   105      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: watchdog/19     S ffff910369536180     0   106      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90d9f75>] ? local_clock+0x25/0x30
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90da029>] ? running_clock+0x9/0x10
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: migration/19    S ffff910369535140     0   107      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa912ff20>] ? cpu_stop_queue_work+0x70/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:20 fir-io2-s2 kernel: ksoftirqd/19    S ffff910369534100     0   108      2 0x00000000
Apr 22 18:51:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: kworker/19:0H   S ffff910369561040     0   110      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: watchdog/20     S ffff90d3e9656180     0   111      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: migration/20    S ffff90d3e9655140     0   112      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: ksoftirqd/20    S ffff90d3e96a0000     0   113      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: kworker/20:0H   S ffff90d3e96a2080     0   115      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: watchdog/21     S ffff90e3698c6180     0   116      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: migration/21    S ffff90e3698c5140     0   117      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: ksoftirqd/21    S ffff90e3698c4100     0   118      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: kworker/21:0H   S ffff90e369911040     0   120      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: watchdog/22     S ffff90f369476180     0   121      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9148c40>] ? touch_nmi_watchdog+0x30/0x30
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: migration/22    S ffff90f369475140     0   122      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: ksoftirqd/22    S ffff90f369474100     0   123      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: kworker/22:0H   S ffff90f369472080     0   125      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: watchdog/23     S ffff910369566180     0   126      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90d9f75>] ? local_clock+0x25/0x30
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90da029>] ? running_clock+0x9/0x10
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: migration/23    S ffff910369565140     0   127      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: ksoftirqd/23    S ffff910369564100     0   128      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: kworker/23:0H   S ffff910369562080     0   130      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: watchdog/24     S ffff90d3e96a6180     0   131      2 0x00000000
Apr 22 18:51:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9148c40>] ? touch_nmi_watchdog+0x30/0x30
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: migration/24    S ffff90d3e96a5140     0   132      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: ksoftirqd/24    S ffff90d3e96a4100     0   133      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: kworker/24:0H   S ffff90d3e9708000     0   135      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: watchdog/25     S ffff90e369916180     0   136      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: migration/25    S ffff90e369915140     0   137      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: ksoftirqd/25    S ffff90e369914100     0   138      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: kworker/25:0H   S ffff90e369912080     0   140      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: watchdog/26     S ffff90f3694b0000     0   141      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: migration/26    S ffff90f3694b1040     0   142      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: ksoftirqd/26    S ffff90f3694b2080     0   143      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: kworker/26:0    S ffff90f3694b30c0     0   144      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90b9f06>] ? process_one_work+0x336/0x440
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: kworker/26:0H   S ffff90f3694b4100     0   145      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: watchdog/27     S ffff9103695c8000     0   146      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90d9f75>] ? local_clock+0x25/0x30
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90da029>] ? running_clock+0x9/0x10
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: migration/27    S ffff9103695c9040     0   147      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa976a4db>] ? _raw_spin_lock_irqsave+0xb/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: ksoftirqd/27    S ffff9103695ca080     0   148      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: kworker/27:0    S ffff9103695cb0c0     0   149      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90b9f06>] ? process_one_work+0x336/0x440
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: kworker/27:0H   S ffff9103695cc100     0   150      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: watchdog/28     S ffff90d3e970e180     0   151      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:22 fir-io2-s2 kernel: migration/28    S ffff90d3e970d140     0   152      2 0x00000000
Apr 22 18:51:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: ksoftirqd/28    S ffff90d3e970c100     0   153      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: kworker/28:0H   S ffff90d3e970a080     0   155      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: watchdog/29     S ffff90e369958000     0   156      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: migration/29    S ffff90e369959040     0   157      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: ksoftirqd/29    S ffff90e36995a080     0   158      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: kworker/29:0    S ffff90e36995b0c0     0   159      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: kworker/29:0H   S ffff90e36995c100     0   160      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: watchdog/30     S ffff90f3694b6180     0   161      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9148df0>] ? watchdog_cleanup+0x10/0x10
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: migration/30    S ffff90f3694b5140     0   162      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa91301e0>] ? cpu_stop_unpark+0x50/0x50
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: ksoftirqd/30    S ffff90f3694e8000     0   163      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: kworker/30:0H   S ffff90f3694ea080     0   165      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: watchdog/31     S ffff9103695ce180     0   166      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: migration/31    S ffff9103695cd140     0   167      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: ksoftirqd/31    S ffff910369600000     0   168      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca562>] smpboot_thread_fn+0xe2/0x1a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ca480>] ? lg_double_unlock+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: kworker/31:0H   S ffff910369602080     0   170      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: kdevtmpfs       S ffff90d3e9002080     0   175      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa94aedbd>] devtmpfsd+0x16d/0x180
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa94aec50>] ? handle_create.isra.2+0x220/0x220
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: netns           S ffff90d3e90030c0     0   176      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:23 fir-io2-s2 kernel: khungtaskd      S ffff90d3e9004100     0   177      2 0x00000000
Apr 22 18:51:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa97657ee>] schedule_timeout_interruptible+0x1e/0x20
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa914893c>] watchdog+0x4c/0x2c0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa91488f0>] ? reset_hung_task_detector+0x20/0x20
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: writeback       S ffff90d3e9005140     0   178      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: kintegrityd     S ffff90d3e9006180     0   179      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: bioset          S ffff90d3e9001040     0   180      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: bioset          S ffff90d3e9000000     0   181      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: bioset          S ffff90e2befa8000     0   182      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: kblockd         S ffff90e2befa9040     0   183      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: md              S ffff90e2befaa080     0   184      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: edac-poller     S ffff90e2befab0c0     0   185      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: watchdogd       S ffff90e2befac100     0   186      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: irq/26-AMD-Vi   S ffff90e2befae180     0   188      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b3d0>] ? irq_finalize_oneshot.part.31+0xf0/0xf0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b6f1>] irq_thread+0xd1/0x180
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b470>] ? irq_forced_thread_fn+0x50/0x50
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b620>] ? irq_thread_check_affinity+0xf0/0xf0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: irq/27-AMD-Vi   S ffff90e2bc418000     0   189      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b3d0>] ? irq_finalize_oneshot.part.31+0xf0/0xf0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b6f1>] irq_thread+0xd1/0x180
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b470>] ? irq_forced_thread_fn+0x50/0x50
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b620>] ? irq_thread_check_affinity+0xf0/0xf0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: irq/28-AMD-Vi   S ffff90e2bc419040     0   190      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b3d0>] ? irq_finalize_oneshot.part.31+0xf0/0xf0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b6f1>] irq_thread+0xd1/0x180
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b470>] ? irq_forced_thread_fn+0x50/0x50
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b620>] ? irq_thread_check_affinity+0xf0/0xf0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: irq/29-AMD-Vi   S ffff90e2bc41a080     0   191      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b3d0>] ? irq_finalize_oneshot.part.31+0xf0/0xf0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b6f1>] irq_thread+0xd1/0x180
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b470>] ? irq_forced_thread_fn+0x50/0x50
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa914b620>] ? irq_thread_check_affinity+0xf0/0xf0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: kswapd0         S ffff90e2bc41b0c0     0   196      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa91d7c77>] ? calculate_normal_threshold+0x27/0x50
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa91ce05f>] kswapd+0x38f/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa91cdcd0>] ? balance_pgdat+0x5e0/0x5e0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: kswapd1         S ffff90e2bc4e0000     0   197      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa91d7c77>] ? calculate_normal_threshold+0x27/0x50
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa91ce05f>] kswapd+0x38f/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa91cdcd0>] ? balance_pgdat+0x5e0/0x5e0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: kswapd2         S ffff90e2bc4e1040     0   198      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa91d7c77>] ? calculate_normal_threshold+0x27/0x50
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa91ce05f>] kswapd+0x38f/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa91cdcd0>] ? balance_pgdat+0x5e0/0x5e0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: kswapd3         S ffff90e2bc4e2080     0   199      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa91d7c77>] ? calculate_normal_threshold+0x27/0x50
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa91ce05f>] kswapd+0x38f/0x440
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa91cdcd0>] ? balance_pgdat+0x5e0/0x5e0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:24 fir-io2-s2 kernel: ksmd            S ffff90e2bc4e30c0     0   200      2 0x00000000
Apr 22 18:51:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9216e0a>] ksm_scan_thread+0x16a/0x240
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa9216ca0>] ? ksm_do_scan+0x11b0/0x11b0
Apr 22 18:51:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: khugepaged      S ffff90e2bc4e4100     0   201      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9229c56>] khugepaged+0x226/0x480
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9229a30>] ? khugepaged_scan_mm_slot+0xf00/0xf00
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: crypto          S ffff90e2bc4e5140     0   202      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: kthrotld        S ffff90e2bc54e180     0   210      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: kmpath_rdacd    S ffff90e2bc54c100     0   215      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: kaluad          S ffff90e2bc54b0c0     0   216      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: kworker/1:2     S ffff90e36995d140     0   217      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: kpsmoused       S ffff90e2bc548000     0   219      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: ipv6_addrconf   S ffff90e2bc5e1040     0   221      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: deferwq         S ffff90e2bc75b0c0     0   234      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: kworker/9:1     S ffff90f2bc768000     0   237      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: kworker/25:1    S ffff90f2bc76e180     0   239      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: kauditd         S ffff90e2bbdb9040     0   341      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9130fc4>] kauditd_thread+0x184/0x1e0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9130e40>] ? audit_printk_skb+0x70/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: kworker/18:1    S ffff9102bc664100     0   607      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: kworker/31:1    S ffff9112bcd7d140     0  1569      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90b9f06>] ? process_one_work+0x336/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: kworker/17:1    S ffff90f2b9486180     0  1571      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: scsi_eh_0       S ffff90f2b9506180     0  1801      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa94d61b3>] scsi_error_handler+0xa3/0x8b0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa94d6110>] ? scsi_eh_get_sense+0x250/0x250
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: scsi_tmf_0      S ffff9102bb4d8c80     0  1806      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: kworker/22:1    S ffff9102b8f40000     0  1927      2 0x00000000
Apr 22 18:51:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90b7467>] ? start_worker+0x27/0x30
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90baba9>] ? manage_workers.isra.25+0x189/0x2a0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: poll_mpt3sas0_h S ffff90f2b7a62080     0  5213      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: scsi_eh_1       S ffff90f2b7482080     0  5512      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa94d61b3>] scsi_error_handler+0xa3/0x8b0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa94d6110>] ? scsi_eh_get_sense+0x250/0x250
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: scsi_tmf_1      S ffff90f2b74830c0     0  5513      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: kworker/19:1    S ffff9112baa2e180     0  5528      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: ata_sff         S ffff9102bc414b00     0  5531      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: mlx5_health0000 S ffff90f2b92cb0c0     0  7057      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: mlx5_cmd_0000:8 S ffff90f2b92ca080     0  7060      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: kworker/2:2     S ffff9102bc486180     0  7061      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: scsi_eh_2       S ffff9102bc482080     0  7062      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa94d0098>] ? __scsi_iterate_devices+0x68/0x90
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa94d61b3>] scsi_error_handler+0xa3/0x8b0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa94d6110>] ? scsi_eh_get_sense+0x250/0x250
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: scsi_tmf_2      S ffff9102bc485140     0  7063      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: kworker/23:1    S ffff9112bcd7c100     0  7117      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90b9f06>] ? process_one_work+0x336/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: mlx5_page_alloc S ffff9102bcf61040     0  7119      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: kworker/u261:0  S ffff90f2b7a60000     0  7187      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: kworker/u262:0  S ffff90e2bc7a2080     0  7188      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: kworker/u263:0  S ffff90f2b7a65140     0  7189      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: kworker/u265:0  S ffff9112bcd7b0c0     0  7191      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90d0884>] ? finish_task_switch+0x54/0x1c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: mlx5_fc         S ffff90f2bb71c100     0  7358      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: ib-comp-wq      S ffff9102bae61040     0  7371      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: ib-comp-unb-wq  S ffff90f2bb5e1040     0  7372      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:26 fir-io2-s2 kernel: ib_mcast        S ffff90f2bb5e30c0     0  7373      2 0x00000000
Apr 22 18:51:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:26 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: ib_nl_sa_wq     S ffff90f2bb5e4100     0  7374      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: mlx5_ib_sigerr_ S ffff90f2bb5e5140     0  7382      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: ib_mad1         S ffff90f2bb5e6180     0  7395      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: to_fifo         S ffff90f2bb5e2080     0  7396      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: mkey_cache      S ffff90f2b92ce180     0  7399      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: kworker/8:1H    S ffff90e2bc7a30c0     0  7545      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: kworker/u264:1  S ffff9102bacc4100     0  7563      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffc043a158>] ? ib_cq_poll_work+0x48/0x70 [ib_core]
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: kworker/0:1H    S ffff90e2bc41c100     0  7580      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: poll_mpt3sas0_s S ffff90f2b7484100     0  9839      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: kworker/24:3    S ffff90e2b6ee4100     0 10637      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: kworker/10:2    S ffff9102b7365140     0 10641      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: kworker/13:2    S ffff90f2b69bb0c0     0 13679      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90b9f06>] ? process_one_work+0x336/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: kworker/21:2    S ffff90f2b69be180     0 13727      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: kworker/28:1H   S ffff90e2b3ff6180     0 13728      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: kworker/30:1H   S ffff91029a5c8000     0 13729      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: kworker/24:1H   S ffff90e2b3ff1040     0 13730      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: kworker/22:1H   S ffff91029a5ce180     0 13731      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: kworker/20:1H   S ffff90e2b3ff5140     0 13732      2 0x00000000
Apr 22 18:51:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/16:1H   S ffff90e2b3ff4100     0 13733      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/14:1H   S ffff91029a5c9040     0 13734      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/12:1H   R  running task        0 13735      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/6:1H    S ffff91029a5cd140     0 13736      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/10:1H   S ffff91029a5cc100     0 13737      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/4:1H    S ffff90e2b3fcb0c0     0 13738      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/2:1H    S ffff91029a5e8000     0 13739      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/31:1H   S ffff9112b7a6e180     0 13740      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/29:1H   S ffff90f2b69bd140     0 13741      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/19:1H   S ffff9112b7a69040     0 13742      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/1:1H    S ffff90f2b81d8000     0 13743      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/26:1H   S ffff91029a5ee180     0 13744      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: kworker/30:2    S ffff9102ba551040     0 14219      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: jbd2/sdif2-8    S ffff90f2b81da080     0 14253      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffc09e9fc3>] kjournald2+0x203/0x260 [jbd2]
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffc09e9dc0>] ? commit_timeout+0x10/0x10 [jbd2]
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: ext4-rsv-conver S ffff90f2b81db0c0     0 14254      2 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:28 fir-io2-s2 kernel: systemd-journal S 0000000000000004     0 14326      1 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa976778f>] ? __schedule+0x3ff/0x890
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9766e7d>] ? schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa928eaa9>] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa925406d>] ? do_filp_open+0x4d/0xb0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9766eb3>] ? schedule_hrtimeout_range+0x13/0x20
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa921bf25>] ? kmem_cache_alloc+0x35/0x1f0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa923f008>] ? do_truncate+0x88/0xc0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa92616f7>] ? __alloc_fd+0x47/0x170
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9252d2d>] ? putname+0x3d/0x60
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa929020d>] ? SyS_epoll_wait+0xed/0x120
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa976f915>] ? do_page_fault+0x35/0x90
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9774ddb>] ? system_call_fastpath+0x22/0x27
Apr 22 18:51:28 fir-io2-s2 kernel: systemd-udevd   S ffff90f2b68e6180     0 14367      1 0x00000000
Apr 22 18:51:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa921bf25>] ? kmem_cache_alloc+0x35/0x1f0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9766e7d>] schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa928eaa9>] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa928ed3e>] ep_poll+0x23e/0x360
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa9253461>] ? do_unlinkat+0xf1/0x2d0
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:51:28 fir-io2-s2 kernel: [<ffffffffa929020d>] SyS_epoll_wait+0xed/0x120
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:29 fir-io2-s2 kernel: kmpathd         S ffff90e2b5e26180     0 17049      2 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:29 fir-io2-s2 kernel: kmpath_handlerd S ffff90e2b58a30c0     0 17051      2 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:29 fir-io2-s2 kernel: multipathd      S ffff9102b966a080     0 17151      1 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9264314>] ? mntput+0x24/0x40
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa92436e6>] ? __fput+0x186/0x260
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90be7a0>] ? task_work_run+0xc0/0xe0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:29 fir-io2-s2 kernel: multipathd      S ffff90e2b9df6180     0 17172      1 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:29 fir-io2-s2 kernel: multipathd      S ffff91029ae80000     0 17199      1 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9766e7d>] schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa921b581>] ? __slab_free+0x81/0x2f0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c273c>] ? add_wait_queue+0x3c/0x50
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9256f6f>] ? __pollwait+0x7f/0xf0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa925884d>] do_sys_poll+0x48d/0x590
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9673112>] ? netlink_recvmsg+0x212/0x490
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa96194f5>] ? sock_recvmsg+0xc5/0x100
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa91656c5>] ? tracing_is_on+0x15/0x30
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9061dee>] ? physflat_send_IPI_mask+0xe/0x10
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90d64f0>] ? try_to_wake_up+0x190/0x390
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90d676b>] ? wake_up_q+0x5b/0x80
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910d525>] ? futex_wake_op+0x4a5/0x610
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910f9da>] ? do_futex+0x17a/0x5a0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa937ae91>] ? plist_add+0x91/0xd0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa902a621>] ? __switch_to+0x151/0x580
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9258a54>] SyS_poll+0x74/0x110
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:29 fir-io2-s2 kernel: multipathd      S ffff91029ae82080     0 17200      1 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa925884d>] do_sys_poll+0x48d/0x590
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa91c06a2>] ? get_page_from_freelist+0x412/0xa70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa91c0e76>] ? __alloc_pages_nodemask+0x176/0x420
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9232c22>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa91c5bc8>] ? add_page_to_unevictable_list+0xc8/0x100
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa91f70ed>] ? page_add_new_anon_rmap+0x12d/0x170
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa91e9668>] ? handle_pte_fault+0x5b8/0xd10
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa921d485>] ? __kmalloc+0x55/0x230
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9258bde>] SyS_ppoll+0xee/0x1f0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:29 fir-io2-s2 kernel: multipathd      S ffff910296bb6180     0 20085      1 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9766c46>] do_nanosleep+0x86/0xf0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c6a6b>] hrtimer_nanosleep+0xbb/0x180
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9766c32>] ? do_nanosleep+0x72/0xf0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c6bc6>] SyS_nanosleep+0x96/0xb0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:29 fir-io2-s2 kernel: multipathd      S ffff910369605140     0 20086      1 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9264314>] ? mntput+0x24/0x40
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910cbf8>] ? get_futex_key+0x1c8/0x2c0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa93874a8>] ? lockref_put_or_lock+0x48/0x80
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9264314>] ? mntput+0x24/0x40
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa92436e6>] ? __fput+0x186/0x260
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90be7a0>] ? task_work_run+0xc0/0xe0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:29 fir-io2-s2 kernel: multipathd      S ffff9112bc7d0000     0 20263      1 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:29 fir-io2-s2 kernel: multipathd      S ffff9112bc7d5140     0 20285      1 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:29 fir-io2-s2 kernel: multipathd      S ffff9112bc7d2080     0 20298      1 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:29 fir-io2-s2 kernel: multipathd      S ffff9112bc7d30c0     0 20311      1 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:29 fir-io2-s2 kernel: multipathd      S ffff9112bc7d1040     0 20345      1 0x00000000
Apr 22 18:51:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:29 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:30 fir-io2-s2 kernel: multipathd      S ffff9112bc7d4100     0 20369      1 0x00000000
Apr 22 18:51:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:30 fir-io2-s2 kernel: multipathd      S ffff9102b8f430c0     0 20422      1 0x00000000
Apr 22 18:51:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:30 fir-io2-s2 kernel: multipathd      S ffff9102b8f45140     0 20443      1 0x00000000
Apr 22 18:51:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:30 fir-io2-s2 kernel: multipathd      S ffff91029c76c100     0 20483      1 0x00000000
Apr 22 18:51:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:30 fir-io2-s2 kernel: multipathd      S ffff91029c76a080     0 20502      1 0x00000000
Apr 22 18:51:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:30 fir-io2-s2 kernel: multipathd      S ffff91029c76b0c0     0 20651      1 0x00000000
Apr 22 18:51:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:30 fir-io2-s2 kernel: multipathd      S ffff9102977d6180     0 20688      1 0x00000000
Apr 22 18:51:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:30 fir-io2-s2 kernel: multipathd      S ffff9112b8862080     0 20716      1 0x00000000
Apr 22 18:51:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:30 fir-io2-s2 kernel: multipathd      S ffff9112b8866180     0 20938      1 0x00000000
Apr 22 18:51:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:30 fir-io2-s2 kernel: multipathd      S ffff9112b8860000     0 20986      1 0x00000000
Apr 22 18:51:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9260ae5>] ? fget_light+0x45/0xe0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:30 fir-io2-s2 kernel: multipathd      S ffff9112b8865140     0 21206      1 0x00000000
Apr 22 18:51:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:31 fir-io2-s2 kernel: multipathd      S ffff910298a55140     0 21222      1 0x00000000
Apr 22 18:51:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:31 fir-io2-s2 kernel: multipathd      S ffff910298a52080     0 21342      1 0x00000000
Apr 22 18:51:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:31 fir-io2-s2 kernel: multipathd      S ffff910298a50000     0 21373      1 0x00000000
Apr 22 18:51:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9232c22>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:31 fir-io2-s2 kernel: multipathd      S ffff9102ba729040     0 21479      1 0x00000000
Apr 22 18:51:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:31 fir-io2-s2 kernel: multipathd      S ffff9112bab2a080     0 21820      1 0x00000000
Apr 22 18:51:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:31 fir-io2-s2 kernel: multipathd      S ffff9112bab2e180     0 21877      1 0x00000000
Apr 22 18:51:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:31 fir-io2-s2 kernel: multipathd      S ffff90f2bb71b0c0     0 22066      1 0x00000000
Apr 22 18:51:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:31 fir-io2-s2 kernel: multipathd      S ffff9102bb640000     0 22188      1 0x00000000
Apr 22 18:51:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:31 fir-io2-s2 kernel: multipathd      S ffff9102bb6430c0     0 22215      1 0x00000000
Apr 22 18:51:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:31 fir-io2-s2 kernel: multipathd      S ffff9102bb645140     0 22266      1 0x00000000
Apr 22 18:51:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:31 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:32 fir-io2-s2 kernel: multipathd      S ffff9102bb646180     0 22371      1 0x00000000
Apr 22 18:51:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:32 fir-io2-s2 kernel: multipathd      S ffff9102bb644100     0 22475      1 0x00000000
Apr 22 18:51:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:32 fir-io2-s2 kernel: multipathd      S ffff9102bb641040     0 22545      1 0x00000000
Apr 22 18:51:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:32 fir-io2-s2 kernel: multipathd      S ffff9112b7a6a080     0 22587      1 0x00000000
Apr 22 18:51:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:32 fir-io2-s2 kernel: multipathd      S ffff9102bece9040     0 22690      1 0x00000000
Apr 22 18:51:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:32 fir-io2-s2 kernel: multipathd      S ffff9102becec100     0 22934      1 0x00000000
Apr 22 18:51:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:32 fir-io2-s2 kernel: multipathd      S ffff9102becea080     0 23005      1 0x00000000
Apr 22 18:51:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:32 fir-io2-s2 kernel: multipathd      S ffff9102bece8000     0 23205      1 0x00000000
Apr 22 18:51:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:32 fir-io2-s2 kernel: multipathd      S ffff9102beceb0c0     0 23451      1 0x00000000
Apr 22 18:51:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:32 fir-io2-s2 kernel: multipathd      S ffff9102becee180     0 23464      1 0x00000000
Apr 22 18:51:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:32 fir-io2-s2 kernel: multipathd      S ffff90f2bb719040     0 23488      1 0x00000000
Apr 22 18:51:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:32 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:33 fir-io2-s2 kernel: multipathd      S ffff9102b87be180     0 23523      1 0x00000000
Apr 22 18:51:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:33 fir-io2-s2 kernel: multipathd      S ffff90f2b92cd140     0 23921      1 0x00000000
Apr 22 18:51:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:33 fir-io2-s2 kernel: multipathd      S ffff90f2b92c8000     0 24074      1 0x00000000
Apr 22 18:51:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:33 fir-io2-s2 kernel: multipathd      S ffff9102beced140     0 24157      1 0x00000000
Apr 22 18:51:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:33 fir-io2-s2 kernel: multipathd      S ffff910299285140     0 24214      1 0x00000000
Apr 22 18:51:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90af86b>] ? __set_current_blocked+0x5b/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:33 fir-io2-s2 kernel: multipathd      S ffff9102b87b9040     0 24990      1 0x00000000
Apr 22 18:51:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:33 fir-io2-s2 kernel: multipathd      S ffff9112b1a08000     0 25526      1 0x00000000
Apr 22 18:51:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:33 fir-io2-s2 kernel: multipathd      S ffff9112b1a0e180     0 25605      1 0x00000000
Apr 22 18:51:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:33 fir-io2-s2 kernel: multipathd      S ffff90f2b738a080     0 25869      1 0x00000000
Apr 22 18:51:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:33 fir-io2-s2 kernel: multipathd      S ffff90f2bbfda080     0 26062      1 0x00000000
Apr 22 18:51:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:33 fir-io2-s2 kernel: multipathd      S ffff90f2bbfdc100     0 26117      1 0x00000000
Apr 22 18:51:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:33 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:34 fir-io2-s2 kernel: multipathd      S ffff9102992830c0     0 26322      1 0x00000000
Apr 22 18:51:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:34 fir-io2-s2 kernel: multipathd      S ffff9102b6e58000     0 26386      1 0x00000000
Apr 22 18:51:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:34 fir-io2-s2 kernel: multipathd      S ffff91029c27c100     0 26418      1 0x00000000
Apr 22 18:51:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:34 fir-io2-s2 kernel: multipathd      S ffff9112b88630c0     0 26454      1 0x00000000
Apr 22 18:51:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:34 fir-io2-s2 kernel: multipathd      S ffff9112b1a0a080     0 26498      1 0x00000000
Apr 22 18:51:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:34 fir-io2-s2 kernel: multipathd      S ffff910290a530c0     0 26710      1 0x00000000
Apr 22 18:51:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:34 fir-io2-s2 kernel: multipathd      S ffff910290a54100     0 26782      1 0x00000000
Apr 22 18:51:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:34 fir-io2-s2 kernel: multipathd      S ffff910290a56180     0 26807      1 0x00000000
Apr 22 18:51:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:34 fir-io2-s2 kernel: multipathd      S ffff910290a50000     0 26868      1 0x00000000
Apr 22 18:51:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:34 fir-io2-s2 kernel: multipathd      S ffff910290a51040     0 26941      1 0x00000000
Apr 22 18:51:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:34 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:35 fir-io2-s2 kernel: multipathd      S ffff910290a52080     0 26990      1 0x00000000
Apr 22 18:51:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:35 fir-io2-s2 kernel: multipathd      S ffff910290a55140     0 27104      1 0x00000000
Apr 22 18:51:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:35 fir-io2-s2 kernel: multipathd      S ffff90e2b81ea080     0 27160      1 0x00000000
Apr 22 18:51:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:35 fir-io2-s2 kernel: multipathd      S ffff910299ffe180     0 27219      1 0x00000000
Apr 22 18:51:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:35 fir-io2-s2 kernel: multipathd      S ffff910299ffa080     0 27252      1 0x00000000
Apr 22 18:51:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:35 fir-io2-s2 kernel: multipathd      S ffff910299ff9040     0 27293      1 0x00000000
Apr 22 18:51:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:35 fir-io2-s2 kernel: multipathd      S ffff91029adf2080     0 27676      1 0x00000000
Apr 22 18:51:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:35 fir-io2-s2 kernel: multipathd      S ffff91029adf4100     0 27773      1 0x00000000
Apr 22 18:51:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9232c22>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:35 fir-io2-s2 kernel: multipathd      S ffff910293af1040     0 27890      1 0x00000000
Apr 22 18:51:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:35 fir-io2-s2 kernel: multipathd      S ffff910293af2080     0 27934      1 0x00000000
Apr 22 18:51:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:35 fir-io2-s2 kernel: multipathd      S ffff910293af30c0     0 27955      1 0x00000000
Apr 22 18:51:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:35 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:36 fir-io2-s2 kernel: multipathd      S ffff910293af5140     0 28004      1 0x00000000
Apr 22 18:51:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:36 fir-io2-s2 kernel: multipathd      S ffff910293af6180     0 28134      1 0x00000000
Apr 22 18:51:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:36 fir-io2-s2 kernel: multipathd      S ffff90f2b853d140     0 28164      1 0x00000000
Apr 22 18:51:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:36 fir-io2-s2 kernel: multipathd      S ffff90f2b8539040     0 28349      1 0x00000000
Apr 22 18:51:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:36 fir-io2-s2 kernel: multipathd      S ffff90f2b521d140     0 28415      1 0x00000000
Apr 22 18:51:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:36 fir-io2-s2 kernel: multipathd      S ffff90f2b521b0c0     0 28492      1 0x00000000
Apr 22 18:51:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:36 fir-io2-s2 kernel: multipathd      S ffff910298a54100     0 28655      1 0x00000000
Apr 22 18:51:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:36 fir-io2-s2 kernel: multipathd      S ffff910298a530c0     0 28676      1 0x00000000
Apr 22 18:51:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:36 fir-io2-s2 kernel: multipathd      S ffff910298a56180     0 29071      1 0x00000000
Apr 22 18:51:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:36 fir-io2-s2 kernel: multipathd      S ffff90f2b8bbd140     0 29131      1 0x00000000
Apr 22 18:51:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:36 fir-io2-s2 kernel: multipathd      S ffff90f2b8bbb0c0     0 29192      1 0x00000000
Apr 22 18:51:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:36 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:37 fir-io2-s2 kernel: multipathd      S ffff90f2b8bb9040     0 29219      1 0x00000000
Apr 22 18:51:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:37 fir-io2-s2 kernel: multipathd      S ffff90f2b8bbe180     0 29241      1 0x00000000
Apr 22 18:51:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:37 fir-io2-s2 kernel: multipathd      S ffff90f2b8bbc100     0 29340      1 0x00000000
Apr 22 18:51:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:37 fir-io2-s2 kernel: multipathd      S ffff90e2b3c26180     0 29484      1 0x00000000
Apr 22 18:51:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:37 fir-io2-s2 kernel: multipathd      S ffff90e2b3c24100     0 29516      1 0x00000000
Apr 22 18:51:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:37 fir-io2-s2 kernel: multipathd      S ffff90e2b3c230c0     0 29548      1 0x00000000
Apr 22 18:51:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:37 fir-io2-s2 kernel: multipathd      S ffff90e2b3c20000     0 29856      1 0x00000000
Apr 22 18:51:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:37 fir-io2-s2 kernel: multipathd      S ffff9102b9eb30c0     0 30048      1 0x00000000
Apr 22 18:51:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:37 fir-io2-s2 kernel: multipathd      S ffff91029c236180     0 30157      1 0x00000000
Apr 22 18:51:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:37 fir-io2-s2 kernel: multipathd      S ffff91029a7a5140     0 30316      1 0x00000000
Apr 22 18:51:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:37 fir-io2-s2 kernel: multipathd      S ffff910298b9d140     0 30344      1 0x00000000
Apr 22 18:51:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:37 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:38 fir-io2-s2 kernel: multipathd      S ffff910298b9e180     0 30534      1 0x00000000
Apr 22 18:51:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:38 fir-io2-s2 kernel: multipathd      S ffff9112b232c100     0 30849      1 0x00000000
Apr 22 18:51:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:38 fir-io2-s2 kernel: multipathd      S ffff9112b2329040     0 30896      1 0x00000000
Apr 22 18:51:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:38 fir-io2-s2 kernel: multipathd      S ffff90e2b708b0c0     0 30997      1 0x00000000
Apr 22 18:51:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:38 fir-io2-s2 kernel: multipathd      S ffff90e2b708a080     0 31042      1 0x00000000
Apr 22 18:51:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:38 fir-io2-s2 kernel: multipathd      S ffff9102ba72d140     0 31216      1 0x00000000
Apr 22 18:51:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:38 fir-io2-s2 kernel: multipathd      S ffff9102980a6180     0 31284      1 0x00000000
Apr 22 18:51:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:38 fir-io2-s2 kernel: multipathd      S ffff91029a74e180     0 31424      1 0x00000000
Apr 22 18:51:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:38 fir-io2-s2 kernel: multipathd      S ffff91029a74c100     0 31586      1 0x00000000
Apr 22 18:51:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:38 fir-io2-s2 kernel: multipathd      S ffff91029a749040     0 31719      1 0x00000000
Apr 22 18:51:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:38 fir-io2-s2 kernel: multipathd      S ffff91029adf1040     0 31922      1 0x00000000
Apr 22 18:51:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:39 fir-io2-s2 kernel: multipathd      S ffff91029adf6180     0 32016      1 0x00000000
Apr 22 18:51:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:39 fir-io2-s2 kernel: multipathd      S ffff910293af4100     0 32176      1 0x00000000
Apr 22 18:51:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:39 fir-io2-s2 kernel: multipathd      S ffff9102b87bc100     0 32288      1 0x00000000
Apr 22 18:51:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9260ae5>] ? fget_light+0x45/0xe0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:39 fir-io2-s2 kernel: multipathd      S ffff910296fe1040     0 32440      1 0x00000000
Apr 22 18:51:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:39 fir-io2-s2 kernel: multipathd      S ffff910296fe4100     0 32622      1 0x00000000
Apr 22 18:51:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:39 fir-io2-s2 kernel: multipathd      S ffff91029adf5140     0 32750      1 0x00000000
Apr 22 18:51:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:39 fir-io2-s2 kernel: multipathd      S ffff9102ba6b5140     0 32904      1 0x00000000
Apr 22 18:51:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:39 fir-io2-s2 kernel: multipathd      S ffff91029c27a080     0 32975      1 0x00000000
Apr 22 18:51:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:39 fir-io2-s2 kernel: multipathd      S ffff9102bcf8f080     0 33074      1 0x00000000
Apr 22 18:51:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9232c22>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:39 fir-io2-s2 kernel: multipathd      S ffff91028bf5e180     0 33249      1 0x00000000
Apr 22 18:51:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:39 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:40 fir-io2-s2 kernel: multipathd      S ffff90e2a936e180     0 33318      1 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:40 fir-io2-s2 kernel: multipathd      S ffff9105d8464100     0 33577      1 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:40 fir-io2-s2 kernel: multipathd      S ffff91028bf28000     0 33639      1 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:40 fir-io2-s2 kernel: multipathd      S ffff910298b98000     0 33751      1 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:40 fir-io2-s2 kernel: multipathd      S ffff910290bdc100     0 33989      1 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:40 fir-io2-s2 kernel: multipathd      S ffff90e2a9369040     0 34779      1 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:40 fir-io2-s2 kernel: multipathd      S ffff91028e6c5140     0 34909      1 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d04e1>] dm_wait_event+0xa1/0xf0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6980>] dev_wait+0x50/0xd0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d70fe>] ctl_ioctl+0x24e/0x550 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d6930>] ? table_deps+0x1a0/0x1a0 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffc07d740e>] dm_ctl_ioctl+0xe/0x20 [dm_mod]
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256480>] do_vfs_ioctl+0x3a0/0x5a0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ace21>] ? __set_task_blocked+0x41/0xa0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90af852>] ? __set_current_blocked+0x42/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9256721>] SyS_ioctl+0xa1/0xc0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90afb14>] ? SyS_rt_sigprocmask+0xc4/0x100
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:51:40 fir-io2-s2 kernel: kdmflush        S ffff9102b9ac8000     0 20236      2 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: bioset          S ffff9112ba5ed140     0 20261      2 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: kdmwork-253:0   S ffff9112ba5ea080     0 20262      2 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: kdmflush        S ffff910292f30000     0 20275      2 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: bioset          S ffff910292f31040     0 20278      2 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: kdmwork-253:1   S ffff910292f32080     0 20279      2 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: kdmflush        S ffff910292f330c0     0 20290      2 0x00000000
Apr 22 18:51:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: bioset          S ffff910292f34100     0 20294      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa976a175>] ? _raw_spin_unlock_irqrestore+0x15/0x20
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmwork-253:2   S ffff910292f35140     0 20295      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmflush        S ffff910292f36180     0 20306      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: bioset          S ffff9102b9acd140     0 20308      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmwork-253:3   S ffff91029aff5140     0 20310      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmflush        S ffff91029ae86180     0 20342      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: bioset          S ffff9102b9ac9040     0 20343      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmwork-253:4   S ffff9102b9acc100     0 20344      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmflush        S ffff9102b9acb0c0     0 20361      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: bioset          S ffff9102ba6b8000     0 20365      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmwork-253:5   S ffff9102bc62a080     0 20366      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmflush        S ffff9102bc62e180     0 20418      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: bioset          S ffff9102bc62d140     0 20419      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmwork-253:6   S ffff9102bc5b5140     0 20420      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmflush        S ffff9102bc5b4100     0 20440      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: bioset          S ffff9102bc5b30c0     0 20441      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmwork-253:7   S ffff9102bc5b2080     0 20442      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmflush        S ffff9102bc5b1040     0 20478      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: bioset          S ffff9102bc5b0000     0 20480      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: kdmwork-253:8   S ffff9102bc5b6180     0 20481      2 0x00000000
Apr 22 18:51:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmflush        S ffff9102b73630c0     0 20491      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: bioset          S ffff9102b7364100     0 20495      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmwork-253:9   S ffff9102b7366180     0 20497      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmflush        S ffff9102b7362080     0 20640      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: bioset          S ffff9102b9dc2080     0 20642      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmwork-253:10  S ffff9102b9dc6180     0 20644      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmflush        S ffff9112bca4e180     0 20680      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: bioset          S ffff9112bca4b0c0     0 20685      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmwork-253:11  S ffff9112bca4d140     0 20686      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmflush        S ffff9102b9669040     0 20710      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: bioset          S ffff9102b966c100     0 20712      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmwork-253:12  S ffff9102b966b0c0     0 20713      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmflush        S ffff9102b966e180     0 20922      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: bioset          S ffff9102b9668000     0 20925      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmwork-253:13  S ffff9102bc629040     0 20928      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmflush        S ffff9102bc62c100     0 20978      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: bioset          S ffff9102bc628000     0 20982      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmwork-253:14  S ffff91029a5ed140     0 20983      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: kdmflush        S ffff9102bf4630c0     0 21202      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: bioset          S ffff9102bf466180     0 21203      2 0x00000000
Apr 22 18:51:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kdmwork-253:15  S ffff90e2b5e3a080     0 21204      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kdmflush        S ffff90e2b5e3d140     0 21218      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: bioset          S ffff90e2b5e3b0c0     0 21219      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kdmwork-253:16  S ffff91029c769040     0 21220      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kdmflush        S ffff90e2b3c21040     0 21335      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: bioset          S ffff90e2b5e230c0     0 21337      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kdmwork-253:17  S ffff90e2b5e20000     0 21338      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kdmflush        S ffff90e2b5e22080     0 21367      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: bioset          S ffff90e2b5e21040     0 21369      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kdmwork-253:18  S ffff91029c27d140     0 21370      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kdmflush        S ffff91029c279040     0 21474      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: bioset          S ffff91029c27e180     0 21476      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kdmwork-253:19  S ffff91029c278000     0 21477      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kworker/27:1H   S ffff9112b8864100     0 21667      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kworker/25:1H   S ffff90f2bbfdd140     0 21719      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kworker/17:1H   S ffff90f2bbfdb0c0     0 21757      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kworker/21:1H   S ffff90f2bbfde180     0 21758      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kworker/23:1H   S ffff9112b8861040     0 21759      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: kdmflush        S ffff9112bcd79040     0 21817      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:43 fir-io2-s2 kernel: bioset          S ffff9102ba72a080     0 21818      2 0x00000000
Apr 22 18:51:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:43 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kdmwork-253:20  S ffff9102ba728000     0 21819      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kworker/7:1H    S ffff9103696030c0     0 21824      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kworker/11:1H   S ffff910369604100     0 21825      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kdmflush        S ffff910369606180     0 21874      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: bioset          S ffff9102ba72b0c0     0 21875      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kdmwork-253:21  S ffff9102ba72c100     0 21876      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kdmflush        S ffff9102b7361040     0 22062      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: bioset          S ffff9102b7360000     0 22064      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kdmwork-253:22  S ffff9102bf460000     0 22065      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kdmflush        S ffff9102bf462080     0 22184      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: bioset          S ffff9102b6e26180     0 22186      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kdmwork-253:23  S ffff910298fa30c0     0 22187      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kdmflush        S ffff9102b9dd8000     0 22211      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: bioset          S ffff9102b9dda080     0 22212      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kdmwork-253:24  S ffff9102b9dd9040     0 22213      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kdmflush        S ffff91029c76d140     0 22263      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: bioset          S ffff9102bb33b0c0     0 22264      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kdmwork-253:25  S ffff9102bb33c100     0 22265      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: kdmflush        S ffff9102b703a080     0 22368      2 0x00000000
Apr 22 18:51:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: bioset          S ffff9102bf461040     0 22369      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmwork-253:26  S ffff91029a5ca080     0 22370      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmflush        S ffff910298a2d140     0 22472      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: bioset          S ffff910298a2e180     0 22473      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmwork-253:27  S ffff910298a2b0c0     0 22474      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmflush        S ffff9102bb339040     0 22542      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: bioset          S ffff91029c76e180     0 22543      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmwork-253:28  S ffff90f3694ec100     0 22544      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmflush        S ffff90d3e97f0000     0 22584      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: bioset          S ffff90d3e97f30c0     0 22585      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmwork-253:29  S ffff90e2ba0e6180     0 22586      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmflush        S ffff910298a2c100     0 22687      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: bioset          S ffff910298a28000     0 22688      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmwork-253:30  S ffff910298a29040     0 22689      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmflush        S ffff90e2ae229040     0 22931      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: bioset          S ffff90e2ae22a080     0 22932      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmwork-253:31  S ffff90e2ae22b0c0     0 22933      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmflush        S ffff910299281040     0 23002      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: bioset          S ffff910299282080     0 23003      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmwork-253:32  S ffff910299284100     0 23004      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:45 fir-io2-s2 kernel: kdmflush        S ffff90f2bb71d140     0 23200      2 0x00000000
Apr 22 18:51:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: bioset          S ffff90f2bb71a080     0 23201      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kdmwork-253:33  S ffff90f2bb718000     0 23202      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kworker/15:1H   S ffff9112bba05140     0 23403      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kdmflush        S ffff9112bba00000     0 23448      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: bioset          S ffff9112bba02080     0 23449      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kdmwork-253:34  S ffff9112bba04100     0 23450      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kdmflush        S ffff9112bba06180     0 23460      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: bioset          S ffff9112bba01040     0 23461      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kdmwork-253:35  S ffff9112bba030c0     0 23463      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kworker/5:1H    S ffff90f2b7486180     0 23476      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kdmflush        S ffff90f2b7481040     0 23484      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: bioset          S ffff9102b87bb0c0     0 23486      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kdmwork-253:36  S ffff9102b87ba080     0 23487      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kworker/3:1H    S ffff9112b1fc8000     0 23499      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kdmflush        S ffff9112b1fc9040     0 23520      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: bioset          S ffff9112b1fca080     0 23521      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kdmwork-253:37  S ffff9112b1fcb0c0     0 23522      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: kdmflush        S ffff910299ffb0c0     0 23917      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: bioset          S ffff910299ff8000     0 23919      2 0x00000000
Apr 22 18:51:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmwork-253:38  S ffff90e2b1b82080     0 23920      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmflush        S ffff9102bb33d140     0 24069      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: bioset          S ffff9102b6e5c100     0 24070      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmwork-253:39  S ffff9112aeb1e180     0 24071      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmflush        S ffff9112bab29040     0 24152      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: bioset          S ffff910294710000     0 24155      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmwork-253:40  S ffff910294711040     0 24156      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmflush        S ffff9112bab2d140     0 24208      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: bioset          S ffff9112bab28000     0 24211      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmwork-253:41  S ffff91029a5cb0c0     0 24212      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmflush        S ffff90f2b8916180     0 24979      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: bioset          S ffff9102977d4100     0 24985      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmwork-253:42  S ffff9102b9a6d140     0 24988      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmflush        S ffff90e2b7cf2080     0 25517      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: bioset          S ffff90e2b99d30c0     0 25523      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmwork-253:43  S ffff9102ba6b30c0     0 25525      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmflush        S ffff9102b9ddb0c0     0 25600      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: bioset          S ffff910296fea080     0 25602      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmwork-253:44  S ffff910296feb0c0     0 25603      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: kdmflush        S ffff9102b704b0c0     0 25866      2 0x00000000
Apr 22 18:51:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: bioset          S ffff910298a2a080     0 25867      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmwork-253:45  S ffff9102b9ace180     0 25868      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmflush        S ffff910292b1c100     0 26054      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: bioset          S ffff910292b1a080     0 26058      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmwork-253:46  S ffff910292b1b0c0     0 26060      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmflush        S ffff910292b18000     0 26114      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: bioset          S ffff910292b1d140     0 26115      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmwork-253:47  S ffff910292b1e180     0 26116      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmflush        S ffff91029c231040     0 26317      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: bioset          S ffff91029c234100     0 26319      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmwork-253:48  S ffff91029c232080     0 26320      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmflush        S ffff910296fe6180     0 26382      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: bioset          S ffff910296fe5140     0 26383      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmwork-253:49  S ffff910296fe2080     0 26384      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmflush        S ffff90f2b7388000     0 26410      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: bioset          S ffff91029c230000     0 26413      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmwork-253:50  S ffff9112b7a21040     0 26414      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmflush        S ffff910294716180     0 26451      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: bioset          S ffff910294714100     0 26452      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: kdmwork-253:51  S ffff910294715140     0 26453      2 0x00000000
Apr 22 18:51:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmflush        S ffff90d2d7072080     0 26493      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: bioset          S ffff9112bae430c0     0 26496      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmwork-253:52  S ffff9112bae46180     0 26497      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmflush        S ffff9112aeb1a080     0 26705      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: bioset          S ffff9112aeb18000     0 26707      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmwork-253:53  S ffff9112aeb1c100     0 26708      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmflush        S ffff9112aeb19040     0 26776      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: bioset          S ffff9112aeb1b0c0     0 26778      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmwork-253:54  S ffff9112b8c41040     0 26780      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmflush        S ffff9112b7a20000     0 26803      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: bioset          S ffff90f2bb6a5140     0 26805      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmwork-253:55  S ffff90f2bb6a4100     0 26806      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmflush        S ffff90f2bb6a30c0     0 26864      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: bioset          S ffff90f2bb6a1040     0 26866      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmwork-253:56  S ffff90f2b7078000     0 26867      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmflush        S ffff90f2b5b4a080     0 26928      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: bioset          S ffff90f2b5b4d140     0 26929      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmwork-253:57  S ffff90f2b5b4c100     0 26932      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: kdmflush        S ffff90f2b5b48000     0 26974      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: bioset          S ffff90f2b5b4b0c0     0 26977      2 0x00000000
Apr 22 18:51:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmwork-253:58  S ffff90f2baf8c100     0 26985      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmflush        S ffff90f2baf8d140     0 27085      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: bioset          S ffff90f2b5b49040     0 27086      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmwork-253:59  S ffff9112b1fcd140     0 27087      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmflush        S ffff90e2b708d140     0 27150      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: bioset          S ffff90e2b708c100     0 27154      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90d0884>] ? finish_task_switch+0x54/0x1c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmwork-253:60  S ffff90d2d70730c0     0 27155      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmflush        S ffff9105d83cea40     0 27210      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: bioset          S ffff90f2b9484100     0 27215      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmwork-253:61  S ffff90f2b9480000     0 27216      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmflush        S ffff90f2b9482080     0 27240      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90d0884>] ? finish_task_switch+0x54/0x1c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: bioset          S ffff90f2b9481040     0 27245      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmwork-253:62  S ffff90f2b94830c0     0 27246      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmflush        S ffff90e2b5e3e180     0 27278      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: bioset          S ffff9112ba594100     0 27282      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmwork-253:63  S ffff9112ba592080     0 27284      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmflush        S ffff9112ba590000     0 27668      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: bioset          S ffff9112ba595140     0 27670      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmwork-253:64  S ffff9112ba5930c0     0 27672      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: kdmflush        S ffff9112baf8c100     0 27766      2 0x00000000
Apr 22 18:51:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: bioset          S ffff9102907ae180     0 27769      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmwork-253:65  S ffff9102907ab0c0     0 27770      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmflush        S ffff9102907a9040     0 27885      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: bioset          S ffff9102907a8000     0 27887      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmwork-253:66  S ffff9102907ac100     0 27888      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmflush        S ffff91029930b0c0     0 27931      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: bioset          S ffff910299309040     0 27932      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmwork-253:67  S ffff910296fec100     0 27933      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmflush        S ffff910296fe9040     0 27951      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: bioset          S ffff910296fe8000     0 27952      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmwork-253:68  S ffff910296fee180     0 27953      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmflush        S ffff9112ba5eb0c0     0 27992      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: bioset          S ffff9112baeba080     0 27998      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmwork-253:69  S ffff9112baebe180     0 27999      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmflush        S ffff9112baeb9040     0 28127      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: bioset          S ffff9112ba5e8000     0 28130      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmwork-253:70  S ffff9112ba5ec100     0 28132      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmflush        S ffff9112ba5e9040     0 28156      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: bioset          S ffff9112baebc100     0 28159      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: kdmwork-253:71  S ffff9112baebb0c0     0 28160      2 0x00000000
Apr 22 18:51:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmflush        S ffff9112b876c100     0 28333      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: bioset          S ffff9112b7a25140     0 28337      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmwork-253:72  S ffff9112b7a230c0     0 28338      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmflush        S ffff9112b7a26180     0 28405      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: bioset          S ffff9102907aa080     0 28410      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmwork-253:73  S ffff9112b876a080     0 28411      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmflush        S ffff90f2b8ace180     0 28486      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: bioset          S ffff90f2b8ac9040     0 28488      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmwork-253:74  S ffff90f2b8acd140     0 28489      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmflush        S ffff90f2b8cb6180     0 28648      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: bioset          S ffff90f2b8cb2080     0 28649      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmwork-253:75  S ffff90f2b8cb5140     0 28651      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmflush        S ffff9112bae42080     0 28670      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: bioset          S ffff9112bae44100     0 28672      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmwork-253:76  S ffff9112bb79b0c0     0 28673      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmflush        S ffff9112baf8d140     0 29068      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: bioset          S ffff9112baf8a080     0 29069      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmwork-253:77  S ffff9112baf8b0c0     0 29070      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmflush        S ffff9112ba541040     0 29126      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: bioset          S ffff9112ba546180     0 29129      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:52 fir-io2-s2 kernel: kdmwork-253:78  S ffff9112ba540000     0 29130      2 0x00000000
Apr 22 18:51:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmflush        S ffff9112ba545140     0 29189      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: bioset          S ffff9112baf88000     0 29190      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmwork-253:79  S ffff9112baf89040     0 29191      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmflush        S ffff9112b0371040     0 29213      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: bioset          S ffff9112b0370000     0 29215      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmwork-253:80  S ffff9112b876b0c0     0 29218      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmflush        S ffff90f2bc77e400     0 29233      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: bioset          S ffff90e2bbb5d140     0 29237      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmwork-253:81  S ffff90f2b85b4100     0 29239      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmflush        S ffff90e2bbb59040     0 29336      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: bioset          S ffff9112baebd140     0 29338      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmwork-253:82  S ffff9112af371040     0 29339      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmflush        S ffff910299ffc100     0 29475      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: bioset          S ffff910299ffd140     0 29478      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmwork-253:83  S ffff90e2b3c25140     0 29480      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmflush        S ffff910290b69040     0 29512      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: bioset          S ffff910290b68000     0 29513      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmwork-253:84  S ffff91029adf0000     0 29514      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: kdmflush        S ffff910290b6a080     0 29544      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:53 fir-io2-s2 kernel: bioset          S ffff910290b6e180     0 29545      2 0x00000000
Apr 22 18:51:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmwork-253:85  S ffff910293af0000     0 29546      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmflush        S ffff910296fed140     0 29847      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: bioset          S ffff910290609040     0 29850      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmwork-253:86  S ffff91029060a080     0 29851      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmflush        S ffff910290608000     0 30042      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: bioset          S ffff91029060c100     0 30043      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmwork-253:87  S ffff91029060e180     0 30044      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmflush        S ffff910298a18000     0 30151      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: bioset          S ffff910298a19040     0 30153      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmwork-253:88  S ffff910298a1b0c0     0 30154      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmflush        S ffff910298a1d140     0 30312      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa976a175>] ? _raw_spin_unlock_irqrestore+0x15/0x20
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: bioset          S ffff910298a1a080     0 30314      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmwork-253:89  S ffff910298a1e180     0 30315      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmflush        S ffff9102980a0000     0 30340      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: bioset          S ffff9102980a1040     0 30342      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmwork-253:90  S ffff9102980a2080     0 30343      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmflush        S ffff9102980a30c0     0 30528      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: bioset          S ffff9102980a4100     0 30529      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa976a175>] ? _raw_spin_unlock_irqrestore+0x15/0x20
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: kdmwork-253:91  S ffff9102980a5140     0 30530      2 0x00000000
Apr 22 18:51:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmflush        S ffff9112bb79e180     0 30843      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: bioset          S ffff90f2b7485140     0 30845      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmwork-253:92  S ffff9112baa29040     0 30846      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmflush        S ffff9102bf465140     0 30892      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: bioset          S ffff90f2b8acb0c0     0 30894      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmwork-253:93  S ffff90f2b8acc100     0 30895      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmflush        S ffff90e2a8f16180     0 30986      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: bioset          S ffff90e2a8f130c0     0 30991      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmwork-253:94  S ffff9112b1a0c100     0 30995      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmflush        S ffff90f2baf8a080     0 31038      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: bioset          S ffff90f2baf89040     0 31039      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmwork-253:95  S ffff90f2baf88000     0 31040      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmflush        S ffff91029670e180     0 31201      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: bioset          S ffff91029670d140     0 31205      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmwork-253:96  S ffff91029670a080     0 31206      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmflush        S ffff9102bae66180     0 31278      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: bioset          S ffff90f2bc28e180     0 31281      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmwork-253:97  S ffff910298a1c100     0 31282      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmflush        S ffff90e2b65b5140     0 31396      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: bioset          S ffff90e2b65b6180     0 31397      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:55 fir-io2-s2 kernel: kdmwork-253:98  S ffff90e2b65b30c0     0 31399      2 0x00000000
Apr 22 18:51:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmflush        S ffff90e2a936a080     0 31574      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: bioset          S ffff90e2a9368000     0 31575      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmwork-253:99  S ffff90e2a936d140     0 31576      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmflush        S ffff9112bc6230c0     0 31706      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: bioset          S ffff9112bc626180     0 31708      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmwork-253:100 S ffff9112bc624100     0 31709      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmflush        S ffff9112b0788000     0 31910      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: bioset          S ffff9112b078b0c0     0 31913      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmwork-253:101 S ffff9112b078e180     0 31914      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmflush        S ffff9112b8bb1040     0 32003      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: bioset          S ffff9112b8bb30c0     0 32005      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmwork-253:102 S ffff9112b8bb2080     0 32006      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmflush        S ffff9112b8bb6180     0 32168      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: bioset          S ffff9112b8bb5140     0 32169      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmwork-253:103 S ffff9112ba542080     0 32170      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmflush        S ffff9112bc621040     0 32279      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: bioset          S ffff9112ba591040     0 32283      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmwork-253:104 S ffff9112af38d140     0 32284      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: kdmflush        S ffff90f2b8bb8000     0 32421      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:56 fir-io2-s2 kernel: bioset          S ffff9112bca4c100     0 32422      2 0x00000000
Apr 22 18:51:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmwork-253:105 S ffff9112b232e180     0 32423      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmflush        S ffff9112b232b0c0     0 32602      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: bioset          S ffff9112b232a080     0 32605      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmwork-253:106 S ffff9112b232d140     0 32606      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmflush        S ffff9112af38c100     0 32744      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: bioset          S ffff910299286180     0 32745      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmwork-253:107 S ffff910299280000     0 32746      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmflush        S ffff9112bce6b840     0 32887      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: bioset          S ffff9105d8462080     0 32897      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmwork-253:108 S ffff9105d84630c0     0 32899      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmflush        S ffff9105d8460000     0 32955      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: bioset          S ffff9102ba6b4100     0 32961      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmwork-253:109 S ffff9102907ad140     0 32970      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmflush        S ffff91028e22b0c0     0 33051      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: bioset          S ffff91028e22c100     0 33060      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmwork-253:110 S ffff91028e22d140     0 33064      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmflush        S ffff91028bf59040     0 33241      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: bioset          S ffff91028bf5c100     0 33246      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmwork-253:111 S ffff91028bf5d140     0 33247      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:57 fir-io2-s2 kernel: kdmflush        S ffff91028bf58000     0 33312      2 0x00000000
Apr 22 18:51:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:57 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: bioset          S ffff91028bf5a080     0 33316      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: kdmwork-253:112 S ffff9102bc6630c0     0 33317      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: kdmflush        S ffff91028ee0d140     0 33557      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: bioset          S ffff91028ee0e180     0 33558      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: kdmwork-253:113 S ffff91028ee08000     0 33565      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: kdmflush        S ffff90e2bbc6b840     0 33605      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: bioset          S ffff9102bc622580     0 33623      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: kdmwork-253:114 S ffff91028ee0a080     0 33625      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: kdmflush        S ffff91028bf2e180     0 33743      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: bioset          S ffff9102bb61e180     0 33747      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: kdmwork-253:115 S ffff91029c27b0c0     0 33748      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: kdmflush        S ffff90e2b131d140     0 33983      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: bioset          S ffff910290bde180     0 33987      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: kdmwork-253:116 S ffff910290bdb0c0     0 33988      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: ccp-1-q2        S ffff90f2bbc9cb00     0 34523      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: ccp-1-q3        S ffff90f2b85d8000     0 34529      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: ccp-1-q4        S ffff9102bc46e400     0 34530      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: ccp-2-q0        S ffff90e2b1b81040     0 34717      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:58 fir-io2-s2 kernel: ccp-2-q1        S ffff90e2b1b830c0     0 34718      2 0x00000000
Apr 22 18:51:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:58 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: ccp-2-q2        S ffff90f2b8cb1040     0 34719      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: kdmflush        S ffff90f2b8cb4100     0 34720      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: bioset          S ffff90f2b8cb30c0     0 34721      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: ccp-2-q3        S ffff90f2b8cb0000     0 34725      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: ccp-2-q4        S ffff90e2b07f30c0     0 34726      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: kdmwork-253:117 S ffff90e2b07f1040     0 34729      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: kdmflush        S ffff90e2a8ad2080     0 34818      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: ccp-3-q2        S ffff90e2baeaabc0     0 34826      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: ccp-3-q3        S ffff90e2baeaabc0     0 34827      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: ccp-3-q4        S ffff90e2b7586180     0 34839      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: bioset          S ffff90e2b7582080     0 34859      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: kdmwork-253:118 S ffff90e2a8b42080     0 34861      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c2067>] kthread_worker_fn+0xf7/0x180
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1f70>] ? kthread_stop+0xf0/0xf0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: kworker/9:1H    S ffff90f2b893e180     0 35024      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: ccp-4-q0        S ffff9112bcede400     0 35026      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9772840>] ? kprobe_flush_task+0xd0/0x170
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: ccp-4-q1        S ffff90f2bc28d140     0 35029      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: ccp-4-q2        S ffff90f2bc289040     0 35030      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: ccp-4-q3        S ffff90f2bc28c100     0 35031      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:51:59 fir-io2-s2 kernel: ccp-4-q4        S ffff90e2baeaabc0     0 35032      2 0x00000000
Apr 22 18:51:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:51:59 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-5-q2        S ffff90f2b85b0000     0 35162      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-5-q3        S ffff9102b9d94b00     0 35163      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-5-q4        S ffff90f2baa9a080     0 35198      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-6-q0        S ffff9102b9ecbe80     0 35446      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-6-q1        S ffff90e2baeaa580     0 35453      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-6-q2        S ffff9112bc552580     0 35464      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-6-q3        S ffff9102b9d976c0     0 35476      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-6-q4        S ffff9112a560a080     0 35479      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-7-q2        S ffff90e2bbc88000     0 35576      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-7-q3        S ffff9105d841abc0     0 35578      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-7-q4        S ffff9112a560d140     0 35580      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-8-q0        S ffff90f2b707e180     0 35779      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-8-q1        S ffff9105d8418000     0 35789      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-8-q2        S ffff90e2bbf9abc0     0 35801      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-8-q3        S ffff9112a6af1040     0 35804      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: ccp-8-q4        S ffff9112a6af5140     0 35807      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9034919>] ? sched_clock+0x9/0x10
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a96e4>] ccp_cmd_queue_thread+0x94/0x2b0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9380>] ? ccp_do_cmd_backlog+0xd0/0xd0 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffc05a9650>] ? ccp_unregister_rng+0x30/0x30 [ccp]
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:00 fir-io2-s2 kernel: kvm-irqfd-clean S ffff9102bcf8f6c0     0 36641      2 0x00000000
Apr 22 18:52:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:01 fir-io2-s2 kernel: kworker/18:1H   S ffff9102947130c0     0 43214      2 0x00000000
Apr 22 18:52:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:01 fir-io2-s2 kernel: kworker/13:1H   S ffff90f2baa9b0c0     0 43215      2 0x00000000
Apr 22 18:52:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:01 fir-io2-s2 kernel: auditd          S ffff91027ea1e180     0 67568      1 0x00000000
Apr 22 18:52:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa928ed3e>] ep_poll+0x23e/0x360
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa929020d>] SyS_epoll_wait+0xed/0x120
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9242154>] ? SyS_write+0xa4/0xf0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:01 fir-io2-s2 kernel: auditd          S ffff90f2a2e64100     0 67572      1 0x00000000
Apr 22 18:52:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9061dee>] ? physflat_send_IPI_mask+0xe/0x10
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa976797a>] ? __schedule+0x5ea/0x890
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa976778f>] ? __schedule+0x3ff/0x890
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffc09eae52>] ? jbd2_complete_transaction+0x52/0xa0 [jbd2]
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9275ef8>] ? do_fsync+0x78/0xb0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:01 fir-io2-s2 kernel: rpciod          S ffff90f2bbdc2580     0 67651      2 0x00000080
Apr 22 18:52:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:01 fir-io2-s2 kernel: xprtiod         S ffff90f2bc77ddc0     0 67668      2 0x00000080
Apr 22 18:52:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:01 fir-io2-s2 kernel: irqbalance      S ffff9112af38e180     0 67898      1 0x00000080
Apr 22 18:52:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766c46>] do_nanosleep+0x86/0xf0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c6a6b>] hrtimer_nanosleep+0xbb/0x180
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766c32>] ? do_nanosleep+0x72/0xf0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c6bc6>] SyS_nanosleep+0x96/0xb0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:01 fir-io2-s2 kernel: polkitd         S ffff9112b0789040     0 67915      1 0x00000080
Apr 22 18:52:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766e7d>] schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c273c>] ? add_wait_queue+0x3c/0x50
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa925884d>] do_sys_poll+0x48d/0x590
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa902a621>] ? __switch_to+0x151/0x580
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90d300f>] ? ttwu_do_activate+0x6f/0x80
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90d64f0>] ? try_to_wake_up+0x190/0x390
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa928b38e>] ? inotify_free_event+0xe/0x10
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa921beb6>] ? kfree+0x106/0x140
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9394d0d>] ? list_del+0xd/0x30
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa928bb1f>] ? inotify_read+0x2ef/0x420
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9292547>] ? eventfd_ctx_read+0x67/0x1e0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9258a54>] SyS_poll+0x74/0x110
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9138da6>] ? __audit_syscall_exit+0x1e6/0x280
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:01 fir-io2-s2 kernel: gmain           S ffffffffa976796e     0 67966      1 0x00000080
Apr 22 18:52:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa91e444a>] ? __do_fault.isra.59+0x8a/0x100
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766e7d>] schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c273c>] ? add_wait_queue+0x3c/0x50
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa925884d>] do_sys_poll+0x48d/0x590
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa937c942>] ? radix_tree_lookup_slot+0x22/0x50
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa91b5cbe>] ? __find_get_page+0x1e/0xa0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa91b8e8e>] ? filemap_fault+0x17e/0x490
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa91e4ae9>] ? do_read_fault.isra.61+0x139/0x1b0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa91e93a4>] ? handle_pte_fault+0x2f4/0xd10
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9292547>] ? eventfd_ctx_read+0x67/0x1e0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9258a54>] SyS_poll+0x74/0x110
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9138da6>] ? __audit_syscall_exit+0x1e6/0x280
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:01 fir-io2-s2 kernel: gdbus           S ffff90f2bc76b0c0     0 67972      1 0x00000080
Apr 22 18:52:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9622f01>] ? __kmalloc_reserve.isra.32+0x31/0x90
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766e7d>] schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c273c>] ? add_wait_queue+0x3c/0x50
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c273c>] ? add_wait_queue+0x3c/0x50
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa925884d>] do_sys_poll+0x48d/0x590
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa96193a6>] ? sock_sendmsg+0xb6/0xf0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa96194f5>] ? sock_recvmsg+0xc5/0x100
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa961a269>] ? ___sys_sendmsg+0x3e9/0x400
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9257053>] ? pollwake+0x73/0x90
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9292547>] ? eventfd_ctx_read+0x67/0x1e0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9258a54>] SyS_poll+0x74/0x110
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9138da6>] ? __audit_syscall_exit+0x1e6/0x280
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:01 fir-io2-s2 kernel: polkitd         S ffff90f2bc76a080     0 67973      1 0x00000080
Apr 22 18:52:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766e7d>] schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90c273c>] ? add_wait_queue+0x3c/0x50
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa925884d>] do_sys_poll+0x48d/0x590
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90db738>] ? __enqueue_entity+0x78/0x80
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90d1f9e>] ? resched_curr+0xae/0xc0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90d2ea0>] ? check_preempt_curr+0x80/0xa0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90d2ed9>] ? ttwu_do_wakeup+0x19/0xe0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90d300f>] ? ttwu_do_activate+0x6f/0x80
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90d64f0>] ? try_to_wake_up+0x190/0x390
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa90d676b>] ? wake_up_q+0x5b/0x80
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa910cd80>] ? futex_wake+0x90/0x180
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9292547>] ? eventfd_ctx_read+0x67/0x1e0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9258a54>] SyS_poll+0x74/0x110
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9138da6>] ? __audit_syscall_exit+0x1e6/0x280
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:01 fir-io2-s2 kernel: JS GC Helper    S ffff90f2bc76c100     0 67974      1 0x00000080
Apr 22 18:52:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa91e93a4>] ? handle_pte_fault+0x2f4/0xd10
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa910cbf8>] ? get_futex_key+0x1c8/0x2c0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:01 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:02 fir-io2-s2 kernel: JS Sour~ Thread S ffffffffa976796e     0 67976      1 0x00000080
Apr 22 18:52:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90d676b>] ? wake_up_q+0x5b/0x80
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910d525>] ? futex_wake_op+0x4a5/0x610
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910cbf8>] ? get_futex_key+0x1c8/0x2c0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa931ac74>] ? yama_task_prctl+0x24/0x110
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa937fbc2>] ? strlcpy+0x42/0x60
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:02 fir-io2-s2 kernel: polkitd         S ffffffffa976796e     0 68003      1 0x00000080
Apr 22 18:52:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766e7d>] schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90c273c>] ? add_wait_queue+0x3c/0x50
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa925884d>] do_sys_poll+0x48d/0x590
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90db738>] ? __enqueue_entity+0x78/0x80
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90d1f9e>] ? resched_curr+0xae/0xc0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90d2ea0>] ? check_preempt_curr+0x80/0xa0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90d2ed9>] ? ttwu_do_wakeup+0x19/0xe0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90d300f>] ? ttwu_do_activate+0x6f/0x80
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90d64f0>] ? try_to_wake_up+0x190/0x390
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90d676b>] ? wake_up_q+0x5b/0x80
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910cd80>] ? futex_wake+0x90/0x180
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9292547>] ? eventfd_ctx_read+0x67/0x1e0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9258a54>] SyS_poll+0x74/0x110
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9138da6>] ? __audit_syscall_exit+0x1e6/0x280
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:02 fir-io2-s2 kernel: dbus-daemon     S ffff9112baa2c100     0 67919      1 0x00000080
Apr 22 18:52:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766e7d>] schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa928eaa9>] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa928ed3e>] ep_poll+0x23e/0x360
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa929020d>] SyS_epoll_wait+0xed/0x120
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:02 fir-io2-s2 kernel: sssd            S ffff9112ba16c100     0 67959      1 0x00000080
Apr 22 18:52:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa925884d>] do_sys_poll+0x48d/0x590
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa96194f5>] ? sock_recvmsg+0xc5/0x100
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa91c0e76>] ? __alloc_pages_nodemask+0x176/0x420
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9101066>] ? ktime_get_ts64+0x66/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9258a54>] SyS_poll+0x74/0x110
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:02 fir-io2-s2 kernel: mdadm           S ffff91129a796180     0 67964      1 0x00000080
Apr 22 18:52:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9257c11>] do_select+0x6d1/0x7c0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa91d82c8>] ? zone_statistics+0x88/0xa0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa91c0e76>] ? __alloc_pages_nodemask+0x176/0x420
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa925c6aa>] ? __d_lookup+0x7a/0x160
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa921c74c>] ? kmem_cache_alloc_trace+0x3c/0x200
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa931da23>] ? ima_get_action+0x23/0x30
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa931cf3e>] ? process_measurement+0x8e/0x250
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa93812cb>] ? string.isra.7+0x3b/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa93829e1>] ? vsnprintf+0x201/0x6a0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9257eff>] core_sys_select+0x1ff/0x340
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa925f2eb>] ? iput+0x3b/0x190
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9281671>] ? __blkdev_put+0x141/0x1a0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa925810a>] SyS_select+0xca/0x120
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:02 fir-io2-s2 kernel: chronyd         S ffff90f2aab42080     0 68014      1 0x00000080
Apr 22 18:52:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9257c11>] do_select+0x6d1/0x7c0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa96b1950>] ? udp_rmem_release+0x40/0x40
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa96193a6>] ? sock_sendmsg+0xb6/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa961a673>] ? ___sys_recvmsg+0x133/0x2d0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9257eff>] core_sys_select+0x1ff/0x340
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9259175>] ? __d_free+0x35/0x40
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9259660>] ? d_free+0x60/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9259c28>] ? __dentry_kill+0x128/0x180
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa925810a>] SyS_select+0xca/0x120
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:02 fir-io2-s2 kernel: gssproxy        S ffff90f2aab44100     0 68025      1 0x00000080
Apr 22 18:52:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766e7d>] schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90db738>] ? __enqueue_entity+0x78/0x80
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa924b463>] ? alloc_pipe_info+0xe3/0x170
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa928ed3e>] ep_poll+0x23e/0x360
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa928f5d0>] ? SyS_epoll_ctl+0x130/0xc80
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa929020d>] SyS_epoll_wait+0xed/0x120
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:02 fir-io2-s2 kernel: gssproxy        S ffff91028c6fe180     0 68029      1 0x00000080
Apr 22 18:52:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa91e924c>] ? handle_pte_fault+0x19c/0xd10
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:02 fir-io2-s2 kernel: gssproxy        S ffff90f2b79bc100     0 68030      1 0x00000080
Apr 22 18:52:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa91e93a4>] ? handle_pte_fault+0x2f4/0xd10
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:02 fir-io2-s2 kernel: gssproxy        S ffff90f2b79b8000     0 68031      1 0x00000080
Apr 22 18:52:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:02 fir-io2-s2 kernel: gssproxy        S ffff90f2b79be180     0 68032      1 0x00000080
Apr 22 18:52:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90de6bc>] ? set_next_entity+0x3c/0xe0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa90de7bf>] ? pick_next_task_fair+0x5f/0x1b0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] ? schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa909e04b>] ? do_exit+0x6bb/0xa40
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:02 fir-io2-s2 kernel: gssproxy        S ffff90f2bbc9ea40     0 68034      1 0x00000080
Apr 22 18:52:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:02 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:03 fir-io2-s2 kernel: sssd_be         S ffff90e2b07f5140     0 68036  67959 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa928ed3e>] ep_poll+0x23e/0x360
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa929020d>] SyS_epoll_wait+0xed/0x120
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:03 fir-io2-s2 kernel: smartd          S ffff91027ea1d140     0 68052      1 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766c46>] do_nanosleep+0x86/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c6a6b>] hrtimer_nanosleep+0xbb/0x180
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766c32>] ? do_nanosleep+0x72/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c6bc6>] SyS_nanosleep+0x96/0xb0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:03 fir-io2-s2 kernel: rpcbind         S ffff90e2a7f5e180     0 68075      1 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa925884d>] do_sys_poll+0x48d/0x590
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa937c942>] ? radix_tree_lookup_slot+0x22/0x50
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa91b5cbe>] ? __find_get_page+0x1e/0xa0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa91b8e8e>] ? filemap_fault+0x17e/0x490
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9258a54>] SyS_poll+0x74/0x110
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:03 fir-io2-s2 kernel: sssd_nss        S ffff91029670c100     0 68080  67959 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa928ed3e>] ep_poll+0x23e/0x360
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9264314>] ? mntput+0x24/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa92436e6>] ? __fput+0x186/0x260
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa929020d>] SyS_epoll_wait+0xed/0x120
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:03 fir-io2-s2 kernel: sssd_pam        S ffff910296709040     0 68081  67959 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa928ed3e>] ep_poll+0x23e/0x360
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa929020d>] SyS_epoll_wait+0xed/0x120
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:03 fir-io2-s2 kernel: systemd-logind  S ffff90f2b738d140     0 68090      1 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766e7d>] schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa928eaa9>] ? ep_scan_ready_list.isra.7+0x1b9/0x1f0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa928ed3e>] ep_poll+0x23e/0x360
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa928f6ed>] ? SyS_epoll_ctl+0x24d/0xc80
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa929020d>] SyS_epoll_wait+0xed/0x120
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:03 fir-io2-s2 kernel: mlx4            S ffff9105d8466180     0 68823      2 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: mlx4_ib         S ffff90f2b521a080     0 68854      2 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: mlx4_ib_mcg     S ffff91028c6fd140     0 68855      2 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: ipoib_flush     S ffff910299308000     0 69373      2 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: mlx5i           S ffff91029930c100     0 69374      2 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: ipoib_wq        S ffff90e2b7581040     0 69538      2 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: rdma_cm         S ffff910290b6b0c0     0 69878      2 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:03 fir-io2-s2 kernel: sshd            S ffff90f2b69b9040     0 71586      1 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa91c9e21>] ? shrink_slab+0x1f1/0x340
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766e7d>] schedule_hrtimeout_range_clock+0x12d/0x150
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c273c>] ? add_wait_queue+0x3c/0x50
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9256f6f>] ? __pollwait+0x7f/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9257c11>] do_select+0x6d1/0x7c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa96241d1>] ? __alloc_skb+0xa1/0x2d0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9232c22>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9375979>] ? cpumask_any_but+0x39/0x50
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa907a718>] ? flush_tlb_page+0x38/0xa0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa91c5afe>] ? lru_cache_add+0xe/0x10
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa91f7078>] ? page_add_new_anon_rmap+0xb8/0x170
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa91e58cb>] ? wp_page_copy.isra.71+0x3ab/0x560
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa91b676b>] ? unlock_page+0x2b/0x30
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa91e7b3e>] ? do_wp_page+0x19e/0x720
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa923e2aa>] ? __check_object_size+0x1ca/0x250
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9257eff>] core_sys_select+0x1ff/0x340
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa91ebedd>] ? handle_mm_fault+0x39d/0x9b0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa925810a>] SyS_select+0xca/0x120
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:03 fir-io2-s2 kernel: collectd        S ffff9112baa28000     0 71587      1 0x00000080
Apr 22 18:52:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9766c46>] do_nanosleep+0x86/0xf0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c6a6b>] hrtimer_nanosleep+0xbb/0x180
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9766c32>] ? do_nanosleep+0x72/0xf0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c6bc6>] SyS_nanosleep+0x96/0xb0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:04 fir-io2-s2 kernel: writer#0        S ffff9102b9aca080     0 71594      1 0x00000080
Apr 22 18:52:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9061dee>] ? physflat_send_IPI_mask+0xe/0x10
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90d64f0>] ? try_to_wake_up+0x190/0x390
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cbf8>] ? get_futex_key+0x1c8/0x2c0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa902a621>] ? __switch_to+0x151/0x580
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:04 fir-io2-s2 kernel: writer#1        S ffff910294712080     0 71595      1 0x00000080
Apr 22 18:52:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90d9ed5>] ? sched_clock_cpu+0x85/0xc0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90d64f0>] ? try_to_wake_up+0x190/0x390
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cbf8>] ? get_futex_key+0x1c8/0x2c0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9264314>] ? mntput+0x24/0x40
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa92436e6>] ? __fput+0x186/0x260
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:04 fir-io2-s2 kernel: writer#2        S ffff9102ba72e180     0 71596      1 0x00000080
Apr 22 18:52:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9061dee>] ? physflat_send_IPI_mask+0xe/0x10
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90d64f0>] ? try_to_wake_up+0x190/0x390
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cbf8>] ? get_futex_key+0x1c8/0x2c0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:04 fir-io2-s2 kernel: writer#3        S ffff9102b87bd140     0 71597      1 0x00000080
Apr 22 18:52:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90d2f75>] ? ttwu_do_wakeup+0xb5/0xe0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90d64f0>] ? try_to_wake_up+0x190/0x390
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cbf8>] ? get_futex_key+0x1c8/0x2c0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:04 fir-io2-s2 kernel: writer#4        S ffff91029a7a4100     0 71598      1 0x00000080
Apr 22 18:52:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90d2f75>] ? ttwu_do_wakeup+0xb5/0xe0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90d64f0>] ? try_to_wake_up+0x190/0x390
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cbf8>] ? get_futex_key+0x1c8/0x2c0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:04 fir-io2-s2 kernel: reader#0        S ffff91029a7a30c0     0 71599      1 0x00000080
Apr 22 18:52:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf14>] ? futex_wait_queue_me+0xa4/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa91f18f7>] ? do_munmap+0x327/0x480
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:04 fir-io2-s2 kernel: reader#1        S ffff91029a7a0000     0 71600      1 0x00000080
Apr 22 18:52:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf14>] ? futex_wait_queue_me+0xa4/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9264314>] ? mntput+0x24/0x40
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa92436e6>] ? __fput+0x186/0x260
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:04 fir-io2-s2 kernel: reader#2        S ffff91029a7a1040     0 71601      1 0x00000080
Apr 22 18:52:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf14>] ? futex_wait_queue_me+0xa4/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa91f18f7>] ? do_munmap+0x327/0x480
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:04 fir-io2-s2 kernel: reader#3        S ffff91029a7a2080     0 71602      1 0x00000080
Apr 22 18:52:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf14>] ? futex_wait_queue_me+0xa4/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa91f18f7>] ? do_munmap+0x327/0x480
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:04 fir-io2-s2 kernel: reader#4        S ffff91029a7a6180     0 71603      1 0x00000080
Apr 22 18:52:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:04 fir-io2-s2 kernel: rsyslogd        S ffff9112baa2b0c0     0 71591      1 0x00000080
Apr 22 18:52:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9257c11>] do_select+0x6d1/0x7c0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa928ef4e>] ? ep_poll_callback+0xee/0x210
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa921f408>] ? __kmalloc_node_track_caller+0x58/0x290
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa96241bd>] ? __alloc_skb+0x8d/0x2d0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90db13e>] ? task_scan_min+0x3e/0x60
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90db187>] ? task_scan_max+0x27/0x40
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90df600>] ? task_numa_fault+0x8d0/0xbb0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa91e901e>] ? do_numa_page+0x1be/0x250
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa91e93c6>] ? handle_pte_fault+0x316/0xd10
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa961d4ba>] ? sock_def_readable+0x3a/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa96f771b>] ? unix_dgram_sendmsg+0x3cb/0x770
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90deb3b>] ? should_numa_migrate_memory+0x5b/0x150
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90df600>] ? task_numa_fault+0x8d0/0xbb0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa91bc8af>] ? zone_watermark_ok+0x1f/0x30
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa92213fb>] ? numamigrate_isolate_page+0x7b/0x170
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9257eff>] core_sys_select+0x1ff/0x340
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa91ebedd>] ? handle_mm_fault+0x39d/0x9b0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa923e2aa>] ? __check_object_size+0x1ca/0x250
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa925810a>] SyS_select+0xca/0x120
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:04 fir-io2-s2 kernel: in:imjournal    R  running task        0 71610      1 0x00000080
Apr 22 18:52:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa92570c5>] poll_schedule_timeout+0x55/0xb0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa925884d>] do_sys_poll+0x48d/0x590
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffc09e22d1>] ? jbd2_journal_dirty_metadata+0xd1/0x260 [jbd2]
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffc09ea111>] ? jbd2_journal_release_jbd_inode+0xf1/0x130 [jbd2]
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90db738>] ? __enqueue_entity+0x78/0x80
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90e235f>] ? enqueue_entity+0x2ef/0xbe0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9250d51>] ? link_path_walk+0x81/0x8b0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9256fe0>] ? __pollwait+0xf0/0xf0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90d2f75>] ? ttwu_do_wakeup+0xb5/0xe0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90d300f>] ? ttwu_do_activate+0x6f/0x80
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90d64f0>] ? try_to_wake_up+0x190/0x390
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa90d676b>] ? wake_up_q+0x5b/0x80
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910d525>] ? futex_wake_op+0x4a5/0x610
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa910f9da>] ? do_futex+0x17a/0x5a0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9258cc3>] SyS_ppoll+0x1d3/0x1f0
Apr 22 18:52:04 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:05 fir-io2-s2 kernel: rs:main Q:Reg   S ffff9112bce6d140     0 71611      1 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa910cf36>] futex_wait_queue_me+0xc6/0x130
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa910dc1b>] futex_wait+0x17b/0x280
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa91b8b57>] ? generic_file_aio_write+0x77/0xa0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffc061c322>] ? ext4_file_write+0xd2/0x1e0 [ext4]
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa910f966>] do_futex+0x106/0x5a0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa910fe80>] SyS_futex+0x80/0x190
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:05 fir-io2-s2 kernel: nfsiod          S ffff910298b9b0c0     0 71656      2 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: nfsv4.1-svc     S ffff910298b99040     0 71697      2 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffc0b143bd>] nfs41_callback_svc+0x15d/0x1c0 [nfsv4]
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffc0b14260>] ? nfs4_callback_svc+0x60/0x60 [nfsv4]
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: crond           S ffff9112baa2d140     0 71727      1 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9766c46>] do_nanosleep+0x86/0xf0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c6a6b>] hrtimer_nanosleep+0xbb/0x180
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9766c32>] ? do_nanosleep+0x72/0xf0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c6bc6>] SyS_nanosleep+0x96/0xb0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:05 fir-io2-s2 kernel: agetty          S ffff9112af372080     0 71789      1 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa93f91c3>] ? bit_cursor+0x663/0x6a0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90b788e>] ? get_work_pool+0x3e/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba50b>] ? flush_work+0x13b/0x190
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9459088>] ? set_room+0x68/0xc0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa945acfc>] n_tty_read+0x27c/0x930
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa97696eb>] ? ldsem_down_read+0x3b/0x220
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9456550>] tty_read+0x90/0x100
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa924117f>] vfs_read+0x9f/0x170
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa924203f>] SyS_read+0x7f/0xf0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:05 fir-io2-s2 kernel: agetty          S ffff9112af38a080     0 71792      1 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90b788e>] ? get_work_pool+0x3e/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba50b>] ? flush_work+0x13b/0x190
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa947b215>] ? serial8250_start_tx+0xa5/0xe0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9459088>] ? set_room+0x68/0xc0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa945acfc>] n_tty_read+0x27c/0x930
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa97696eb>] ? ldsem_down_read+0x3b/0x220
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9456550>] tty_read+0x90/0x100
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa924117f>] vfs_read+0x9f/0x170
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa924203f>] SyS_read+0x7f/0xf0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:05 fir-io2-s2 kernel: master          S ffff9112bae40000     0 71947      1 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa928ed3e>] ep_poll+0x23e/0x360
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa929020d>] SyS_epoll_wait+0xed/0x120
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:05 fir-io2-s2 kernel: qmgr            S ffff910298b9a080     0 72539  71947 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c5cad>] ? hrtimer_start_range_ns+0x1ed/0x3c0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9766e02>] schedule_hrtimeout_range_clock+0xb2/0x150
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c57e0>] ? hrtimer_get_res+0x50/0x50
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9766df6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9766eb3>] schedule_hrtimeout_range+0x13/0x20
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa928ed3e>] ep_poll+0x23e/0x360
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9264314>] ? mntput+0x24/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa929020d>] SyS_epoll_wait+0xed/0x120
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774ddb>] system_call_fastpath+0x22/0x27
Apr 22 18:52:05 fir-io2-s2 kernel: kworker/27:2    S ffff9112b1a0b0c0     0 72813      2 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: kworker/6:3     S ffff91029c2330c0     0 72822      2 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: raid5wq         S ffff91028c6fc100     0 72939      2 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: bioset          S ffff9112b8c45140     0 72941      2 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: bioset          S ffff9112b8c44100     0 72942      2 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: md7_raid6       D ffff9112b8c42080     0 72945      2 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa937a9c5>] percpu_ref_switch_to_atomic_sync+0x65/0xb0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa958c2b7>] set_in_sync+0x67/0xe0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa959403f>] md_check_recovery+0x27f/0x500
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffc0bd83b6>] raid5d+0x56/0x670 [raid456]
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9765670>] ? schedule_timeout+0x170/0x2d0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa958b7fd>] md_thread+0x16d/0x1e0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa958b690>] ? find_pers+0x80/0x80
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: bioset          S ffff91028bf2c100     0 73010      2 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: bioset          S ffff91028bf2d140     0 73011      2 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:05 fir-io2-s2 kernel: md11_raid6      D ffff91029060b0c0     0 73031      2 0x00000080
Apr 22 18:52:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa937a9c5>] percpu_ref_switch_to_atomic_sync+0x65/0xb0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958c2b7>] set_in_sync+0x67/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa959403f>] md_check_recovery+0x27f/0x500
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffc0bd83b6>] raid5d+0x56/0x670 [raid456]
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9765670>] ? schedule_timeout+0x170/0x2d0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958b7fd>] md_thread+0x16d/0x1e0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958b690>] ? find_pers+0x80/0x80
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: bioset          S ffff9102b9ddc100     0 73283      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: bioset          S ffff9102b9ddd140     0 73284      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: md1_raid6       D ffff91129a790000     0 73317      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa937a9c5>] percpu_ref_switch_to_atomic_sync+0x65/0xb0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958c2b7>] set_in_sync+0x67/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa959403f>] md_check_recovery+0x27f/0x500
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffc0bd83b6>] raid5d+0x56/0x670 [raid456]
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9765670>] ? schedule_timeout+0x170/0x2d0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958b7fd>] md_thread+0x16d/0x1e0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958b690>] ? find_pers+0x80/0x80
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: bioset          S ffff91028e6c0000     0 73390      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: bioset          S ffff91028e6c30c0     0 73391      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: md3_raid6       D ffff91027969e180     0 73427      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa937a9c5>] percpu_ref_switch_to_atomic_sync+0x65/0xb0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958c2b7>] set_in_sync+0x67/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa959403f>] md_check_recovery+0x27f/0x500
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffc0bd83b6>] raid5d+0x56/0x670 [raid456]
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9765670>] ? schedule_timeout+0x170/0x2d0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958b7fd>] md_thread+0x16d/0x1e0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958b690>] ? find_pers+0x80/0x80
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: bioset          S ffff9102bb638000     0 73618      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: bioset          S ffff9102bb63e180     0 73619      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: md5_raid6       D ffff91029a74a080     0 73678      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa937a9c5>] percpu_ref_switch_to_atomic_sync+0x65/0xb0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958c2b7>] set_in_sync+0x67/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa959403f>] md_check_recovery+0x27f/0x500
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffc0bd83b6>] raid5d+0x56/0x670 [raid456]
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9765670>] ? schedule_timeout+0x170/0x2d0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958b7fd>] md_thread+0x16d/0x1e0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958b690>] ? find_pers+0x80/0x80
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: bioset          S ffff9102b7039040     0 73837      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: bioset          S ffff91028e6c4100     0 73838      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: md9_raid6       D ffff91028e6c6180     0 73942      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa937a9c5>] percpu_ref_switch_to_atomic_sync+0x65/0xb0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958c2b7>] set_in_sync+0x67/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa959403f>] md_check_recovery+0x27f/0x500
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffc0bd83b6>] raid5d+0x56/0x670 [raid456]
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9765670>] ? schedule_timeout+0x170/0x2d0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958b7fd>] md_thread+0x16d/0x1e0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa958b690>] ? find_pers+0x80/0x80
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: kworker/u264:0  S ffff91027969c100     0 78622      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffc043a158>] ? ib_cq_poll_work+0x48/0x70 [ib_core]
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: cfs_rh_00       S ffff90e2b7089040     0 81192      2 0x00000080
Apr 22 18:52:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffc0bffbec>] cfs_wi_scheduler+0x30c/0x460 [libcfs]
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffc0bff8e0>] ? cfs_wi_sched_create+0x680/0x680 [libcfs]
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: cfs_rh_01       S ffff90e2b708e180     0 81193      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc0bffbec>] cfs_wi_scheduler+0x30c/0x460 [libcfs]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc0bff8e0>] ? cfs_wi_sched_create+0x680/0x680 [libcfs]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: cfs_rh_02       S ffff9102ba554100     0 81194      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc0bffbec>] cfs_wi_scheduler+0x30c/0x460 [libcfs]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc0bff8e0>] ? cfs_wi_sched_create+0x680/0x680 [libcfs]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: cfs_rh_03       S ffff9102ba550000     0 81195      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc0bffbec>] cfs_wi_scheduler+0x30c/0x460 [libcfs]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc0bff8e0>] ? cfs_wi_sched_create+0x680/0x680 [libcfs]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: clio            S ffff9102ba555140     0 81228      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr00_000 S ffff9102ba552080     0 81234      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr00_001 S ffff9102ba5530c0     0 81235      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr00_002 S ffff9102ba556180     0 81236      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr00_003 S ffff9102bf464100     0 81237      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr01_000 S ffff91028c6f9040     0 81238      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr01_001 S ffff91028c6f8000     0 81239      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr01_002 S ffff91027ea1b0c0     0 81240      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr01_003 S ffff91027ea1a080     0 81242      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr02_000 S ffff91027ea18000     0 81243      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr02_001 S ffff91027ea1c100     0 81244      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr02_002 S ffff91027ea19040     0 81245      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr02_003 S ffff9102b9a6e180     0 81246      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:07 fir-io2-s2 kernel: ptlrpc_hr03_000 S ffff9102b9a6c100     0 81247      2 0x00000080
Apr 22 18:52:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:07 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: ptlrpc_hr03_001 S ffff9102b9a69040     0 81248      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: ptlrpc_hr03_002 S ffff9102b9a6b0c0     0 81249      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: ptlrpc_hr03_003 S ffff9102b9a6a080     0 81250      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc1248335>] ptlrpc_hr_main+0x135/0x2c0 [ptlrpc]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc1248200>] ? ptlrpc_handle_rs+0x5f0/0x5f0 [ptlrpc]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: kiblnd_connd    S ffff9102b9a68000     0 81266      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1a97c>] kiblnd_connd+0x60c/0xa00 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1a370>] ? kiblnd_check_conns+0x870/0x870 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: kiblnd_sd_00_00 S ffff91029a74d140     0 81268      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: kiblnd_sd_00_01 S ffff91029a748000     0 81269      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: kiblnd_sd_00_02 S ffff91029a74b0c0     0 81270      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: kiblnd_sd_00_03 S ffff91029ae84100     0 81271      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: kiblnd_sd_01_00 S ffff91029ae830c0     0 81272      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d15223>] ? kiblnd_handle_rx+0x213/0x6b0 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] ? schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] ? kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] ? kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ? ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: kiblnd_sd_01_01 S ffff90f2bf7dab80     0 81273      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: kiblnd_sd_01_02 S ffff91029ae81040     0 81274      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1060d>] ? kiblnd_tx_done+0x10d/0x3e0 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: kiblnd_sd_01_03 S ffff91029a5ec100     0 81275      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: kiblnd_sd_02_00 S ffff91029a5ea080     0 81276      2 0x00000080
Apr 22 18:52:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: kiblnd_sd_02_01 S ffff91029a5e9040     0 81277      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: kiblnd_sd_02_02 S ffff90f369f9a080     0 81278      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: kiblnd_sd_02_03 S ffff90f369f9b0c0     0 81279      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: kiblnd_sd_03_00 S ffff90f369f98000     0 81280      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: kiblnd_sd_03_01 S ffff90f3694eb0c0     0 81281      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: kiblnd_sd_03_02 S ffff90f3694ee180     0 81282      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: kiblnd_sd_03_03 S ffff90f3694ed140     0 81283      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b52c>] kiblnd_scheduler+0x32c/0x1180 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0d1b200>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: monitor_thread  S ffff90f3694e9040     0 81284      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c27f6>] ? prepare_to_wait+0x56/0x90
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0c938f2>] lnet_monitor_thread+0x492/0x560 [lnet]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0c93460>] ? lnet_recover_peer_nis+0x670/0x670 [lnet]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: lnet_discovery  S ffff9102bae60000     0 81992      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c27f6>] ? prepare_to_wait+0x56/0x90
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0ca6abf>] lnet_peer_discovery+0xdf/0x1150 [lnet]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc0ca69e0>] ? lnet_peer_merge_data+0xde0/0xde0 [lnet]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: ptlrpcd_rcv     S ffff9102bae64100     0 81993      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: ptlrpcd_00_00   S ffff9102bae62080     0 81994      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: ptlrpcd_00_01   S ffff9102bae65140     0 81995      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: ptlrpcd_00_02   S ffff9102bc7c4100     0 81996      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: ptlrpcd_00_03   S ffff9102bc7c1040     0 81997      2 0x00000080
Apr 22 18:52:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_00_04   S ffff9102bc7c5140     0 81998      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_00_05   S ffff9102bc7c2080     0 81999      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_00_06   S ffff9102bc7c6180     0 82000      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_00_07   S ffff9102bc7c30c0     0 82001      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_01_00   S ffff910296bb2080     0 82002      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_01_01   S ffff910296bb5140     0 82003      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_01_02   S ffff910296bb30c0     0 82004      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_01_03   S ffff910296bb1040     0 82005      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_01_04   S ffff910296bb0000     0 82006      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_01_05   S ffff910296bb4100     0 82007      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_01_06   S ffff9102b703d140     0 82008      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_01_07   S ffff9102b703c100     0 82009      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_02_00   S ffff9102b703b0c0     0 82010      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_02_01   S ffff9102b703e180     0 82011      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_02_02   S ffff9102b7038000     0 82012      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: ptlrpcd_02_03   S ffff9102b9dde180     0 82013      2 0x00000080
Apr 22 18:52:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: ptlrpcd_02_04   S ffff910290b6d140     0 82014      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: ptlrpcd_02_05   S ffff910290b6c100     0 82015      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: ptlrpcd_02_06   S ffff9102977d1040     0 82016      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: ptlrpcd_02_07   S ffff9102977d30c0     0 82017      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: ptlrpcd_03_00   S ffff9102977d5140     0 82018      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: ptlrpcd_03_01   S ffff9102977d2080     0 82019      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: ptlrpcd_03_02   S ffff9102977d0000     0 82020      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: ptlrpcd_03_03   S ffff9102bc62b0c0     0 82021      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: ptlrpcd_03_04   S ffff90f3694730c0     0 82022      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: ptlrpcd_03_05   S ffff90f369470000     0 82023      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: ptlrpcd_03_06   S ffff9102ba6ba080     0 82024      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: ptlrpcd_03_07   S ffff9102ba6b9040     0 82025      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc1057ca4>] ? lu_env_refill+0x24/0x30 [obdclass]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125ea6e>] ptlrpcd+0x2ee/0x550 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffc125e780>] ? ptlrpcd_check+0x590/0x590 [ptlrpc]
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: kworker/11:0    S ffff9112b078a080     0 82108      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: kworker/14:1    S ffff9102ba6be180     0 82110      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90b9f06>] ? process_one_work+0x336/0x440
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: kworker/16:4    S ffff90d3e97f2080     0 82151      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90b9dea>] ? process_one_work+0x21a/0x440
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90bae99>] worker_thread+0x1d9/0x3c0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:11 fir-io2-s2 kernel: kmmpd-md9       D ffff9112b8769040     0 82238      2 0x00000080
Apr 22 18:52:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa958d330>] ? md_handle_request+0xd0/0x150
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:11 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9278f0a>] __wait_on_buffer+0x2a/0x30
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc15d939d>] write_mmp_block+0x11d/0x160 [ldiskfs]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc15d95f8>] kmmpd+0x1a8/0x430 [ldiskfs]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc15d9450>] ? __dump_mmp_msg+0x70/0x70 [ldiskfs]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: jbd2/md9-8      D ffff91029c768000     0 82239      2 0x00000080
Apr 22 18:52:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa91581ef>] ? delayacct_end+0x8f/0xb0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9278f0a>] __wait_on_buffer+0x2a/0x30
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc09e3f02>] jbd2_journal_commit_transaction+0xe82/0x19b0 [jbd2]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc09e9e89>] kjournald2+0xc9/0x260 [jbd2]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc09e9dc0>] ? commit_timeout+0x10/0x10 [jbd2]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: ldiskfs-rsv-con S ffff91028e229040     0 82260      2 0x00000080
Apr 22 18:52:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: kmmpd-md1       D ffff91028e22a080     0 82277      2 0x00000080
Apr 22 18:52:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa958d330>] ? md_handle_request+0xd0/0x150
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9278f0a>] __wait_on_buffer+0x2a/0x30
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc15d939d>] write_mmp_block+0x11d/0x160 [ldiskfs]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc15d95f8>] kmmpd+0x1a8/0x430 [ldiskfs]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc15d9450>] ? __dump_mmp_msg+0x70/0x70 [ldiskfs]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: jbd2/md1-8      D ffff91028e228000     0 82283      2 0x00000080
Apr 22 18:52:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc09e32bc>] jbd2_journal_commit_transaction+0x23c/0x19b0 [jbd2]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa976778f>] ? __schedule+0x3ff/0x890
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90aa0de>] ? try_to_del_timer_sync+0x5e/0x90
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc09e9e89>] kjournald2+0xc9/0x260 [jbd2]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc09e9dc0>] ? commit_timeout+0x10/0x10 [jbd2]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: kmmpd-md3       D ffff91029b33e180     0 82284      2 0x00000080
Apr 22 18:52:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa958d330>] ? md_handle_request+0xd0/0x150
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9278f0a>] __wait_on_buffer+0x2a/0x30
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc15d939d>] write_mmp_block+0x11d/0x160 [ldiskfs]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc15d95f8>] kmmpd+0x1a8/0x430 [ldiskfs]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc15d9450>] ? __dump_mmp_msg+0x70/0x70 [ldiskfs]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: jbd2/md3-8      D ffff91029b33c100     0 82288      2 0x00000080
Apr 22 18:52:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc09e32bc>] jbd2_journal_commit_transaction+0x23c/0x19b0 [jbd2]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa976778f>] ? __schedule+0x3ff/0x890
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90aa0de>] ? try_to_del_timer_sync+0x5e/0x90
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc09e9e89>] kjournald2+0xc9/0x260 [jbd2]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc09e9dc0>] ? commit_timeout+0x10/0x10 [jbd2]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: ldiskfs-rsv-con S ffff91029b339040     0 82302      2 0x00000080
Apr 22 18:52:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: ldiskfs-rsv-con S ffff91029b33d140     0 82308      2 0x00000080
Apr 22 18:52:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: kmmpd-md5       D ffff90e2b99d1040     0 82340      2 0x00000080
Apr 22 18:52:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa958d330>] ? md_handle_request+0xd0/0x150
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9278f0a>] __wait_on_buffer+0x2a/0x30
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc15d939d>] write_mmp_block+0x11d/0x160 [ldiskfs]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc15d95f8>] kmmpd+0x1a8/0x430 [ldiskfs]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc15d9450>] ? __dump_mmp_msg+0x70/0x70 [ldiskfs]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:12 fir-io2-s2 kernel: jbd2/md5-8      D ffff90e2a8ad0000     0 82341      2 0x00000080
Apr 22 18:52:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc09e32bc>] jbd2_journal_commit_transaction+0x23c/0x19b0 [jbd2]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa976778f>] ? __schedule+0x3ff/0x890
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90aa0de>] ? try_to_del_timer_sync+0x5e/0x90
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffc09e9e89>] kjournald2+0xc9/0x260 [jbd2]
Apr 22 18:52:12 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc09e9dc0>] ? commit_timeout+0x10/0x10 [jbd2]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: ldlm_cb00_000   S ffff90e2a8ad5140     0 82347      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: lc_watchdogd    S ffff90e2a8ad1040     0 82349      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc0bf80e9>] lcw_dispatch_main+0x749/0x940 [libcfs]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc0bf79a0>] ? lc_watchdog_add+0x790/0x790 [libcfs]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: ldlm_cb00_001   S ffff91029aff2080     0 82350      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: ldlm_cb01_000   S ffff91029aff6180     0 82351      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: ldlm_cb01_001   S ffff91029aff0000     0 82352      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: ldlm_cb02_000   S ffff91029aff4100     0 82353      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: ldlm_cb02_001   S ffff910298fa2080     0 82354      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: ldlm_cb03_000   S ffff910298fa5140     0 82355      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: ldlm_cb03_001   S ffff910298fa0000     0 82356      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: ldlm_cn00_000   S ffff910298fa4100     0 82357      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa927918d>] __getblk+0x2d/0x300
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc15adda7>] ? ldiskfs_bread+0x27/0xc0 [ldiskfs]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9765721>] ? schedule_timeout+0x221/0x2d0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa92790fc>] ? __find_get_block+0xbc/0x120
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc0d11e63>] ? kiblnd_check_sends_locked+0x2a3/0xe40 [ko2iblnd]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc0d12bd4>] ? kiblnd_launch_tx+0x184/0xc20 [ko2iblnd]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc0d139c7>] ? kiblnd_send+0x357/0xa20 [ko2iblnd]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc0c890e4>] ? lnet_ni_send+0x44/0xd0 [lnet]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc0c90d92>] ? lnet_send+0x82/0x1c0 [lnet]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc0c9119c>] ? LNetPut+0x2cc/0xb50 [lnet]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa921b581>] ? __slab_free+0x81/0x2f0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90e060c>] ? dequeue_entity+0x11c/0x5e0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90de6bc>] ? set_next_entity+0x3c/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa91656c5>] ? tracing_is_on+0x15/0x30
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa91717bd>] ? probe_sched_switch+0x2d/0xa0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] ? schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa976a175>] ? _raw_spin_unlock_irqrestore+0x15/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] ? kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ? ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: ldlm_cn00_001   S ffff910298fa6180     0 82358      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: ldlm_cn00_002   S ffff910298fa1040     0 82359      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:13 fir-io2-s2 kernel: ldlm_cn01_000   S ffff91029670b0c0     0 82360      2 0x00000080
Apr 22 18:52:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ldlm_cn01_001   S ffff910296708000     0 82361      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ldlm_cn01_002   S ffff91028daae180     0 82362      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ldlm_cn02_000   S ffff91028daad140     0 82363      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ldlm_cn02_001   S ffff91028daac100     0 82364      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ldlm_cn02_002   S ffff91028daaa080     0 82365      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ldlm_cn03_000   S ffff91028daab0c0     0 82366      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ldlm_cn03_001   S ffff9102b966d140     0 82367      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ldlm_cn03_002   S ffff910292b19040     0 82368      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ldlm_bl_01      S ffff91029930a080     0 82369      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9153c9d>] ? call_rcu_sched+0x1d/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1023e51>] ? obd_stale_export_get+0x31/0x190 [obdclass]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc121b82d>] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc121b400>] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ldlm_bl_02      S ffff9102b6e5e180     0 82370      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9153c9d>] ? call_rcu_sched+0x1d/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1023e51>] ? obd_stale_export_get+0x31/0x190 [obdclass]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc121b82d>] ldlm_bl_thread_main+0x42d/0x700 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc121b400>] ? ldlm_handle_bl_callback+0x530/0x530 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ldlm_elt        S ffff9102b6e5a080     0 82371      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1213532>] expired_lock_main+0x1c2/0xc00 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1213370>] ? ldlm_server_glimpse_ast+0x3f0/0x3f0 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ll_cfg_requeue  S ffff91028cada080     0 82373      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc163ec65>] mgc_requeue_thread+0x5f5/0x870 [mgc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc163e670>] ? mgc_process_config+0x13f0/0x13f0 [mgc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: ll_ost00_000    D ffff91028cadb0c0     0 82382      2 0x00000080
Apr 22 18:52:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: ll_ost00_001    D ffff91028cadd140     0 82383      2 0x00000080
Apr 22 18:52:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: ll_ost00_002    S ffff91028cad8000     0 82384      2 0x00000080
Apr 22 18:52:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: ll_ost01_000    S ffff91028cadc100     0 82385      2 0x00000080
Apr 22 18:52:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: ll_ost01_001    S ffff91028cad9040     0 82386      2 0x00000080
Apr 22 18:52:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: ll_ost01_002    S ffff910296fe0000     0 82387      2 0x00000080
Apr 22 18:52:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: ll_ost02_000    S ffff910296fe30c0     0 82388      2 0x00000080
Apr 22 18:52:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1209d70>] ldlm_completion_ast+0x430/0x890 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1209720>] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc11f0dcf>] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1207ef2>] ? ldlm_interval_alloc+0xa2/0x260 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc120ac8c>] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc18016ad>] ofd_destroy_by_fid+0x1dd/0x4a0 [ofd]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: ll_ost02_001    D ffff910279699040     0 82390      2 0x00000080
Apr 22 18:52:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: ll_ost02_002    D ffff910279698000     0 82391      2 0x00000080
Apr 22 18:52:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:15 fir-io2-s2 kernel: ll_ost03_000    S ffff91027969a080     0 82392      2 0x00000080
Apr 22 18:52:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:15 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: ll_ost03_001    S ffff9102bc4830c0     0 82393      2 0x00000080
Apr 22 18:52:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: ll_ost03_002    S ffff9102b9dc4100     0 82394      2 0x00000080
Apr 22 18:52:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: ll_ost_create00 S ffff9102b9dc5140     0 82395      2 0x00000080
Apr 22 18:52:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: ll_ost_create00 S ffff9102b9dc0000     0 82396      2 0x00000080
Apr 22 18:52:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: ll_ost_create01 D ffff9102b9dc1040     0 82397      2 0x00000080
Apr 22 18:52:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: ll_ost_create01 S ffff9102b9dc30c0     0 82398      2 0x00000080
Apr 22 18:52:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: ll_ost_create02 D ffff9102ba6bc100     0 82399      2 0x00000080
Apr 22 18:52:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:16 fir-io2-s2 kernel: Lustre: fir-OST0011: Connection restored to cebdfa64-1951-9c8b-f9b2-40379187011e (at 10.8.1.33@o2ib6)
Apr 22 18:52:16 fir-io2-s2 kernel: Lustre: Skipped 100 previous similar messages
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: ll_ost_create02 D ffff9102b6e24100     0 82400      2 0x00000080
Apr 22 18:52:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: ll_ost_create03 S ffff9102b6e20000     0 82401      2 0x00000080
Apr 22 18:52:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: ll_ost_create03 S ffff9102b6e25140     0 82402      2 0x00000080
Apr 22 18:52:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:16 fir-io2-s2 kernel: ll_ost_io00_000 D ffff9102b6e22080     0 82403      2 0x00000080
Apr 22 18:52:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:16 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:17 fir-io2-s2 kernel: ll_ost_io00_001 D ffff9102b6e230c0     0 82404      2 0x00000080
Apr 22 18:52:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:17 fir-io2-s2 kernel: ll_ost_io00_002 D ffff9102b6e21040     0 82405      2 0x00000080
Apr 22 18:52:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:17 fir-io2-s2 kernel: ll_ost_io01_000 D ffff9102b9eb6180     0 82406      2 0x00000080
Apr 22 18:52:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:17 fir-io2-s2 kernel: ll_ost_io01_001 D ffff9102b9eb0000     0 82407      2 0x00000080
Apr 22 18:52:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:17 fir-io2-s2 kernel: ll_ost_io01_002 D ffff9102b9eb4100     0 82408      2 0x00000080
Apr 22 18:52:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:52:17 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:18 fir-io2-s2 kernel: ll_ost_io02_000 D ffff9102b9eb5140     0 82409      2 0x00000080
Apr 22 18:52:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:18 fir-io2-s2 kernel: ll_ost_io02_001 D ffff9102b9eb1040     0 82410      2 0x00000080
Apr 22 18:52:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:18 fir-io2-s2 kernel: ll_ost_io02_002 D ffff9102b9eb2080     0 82411      2 0x00000080
Apr 22 18:52:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:18 fir-io2-s2 kernel: ll_ost_io03_000 D ffff91028bf5b0c0     0 82412      2 0x00000080
Apr 22 18:52:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:18 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: ll_ost_io03_001 D ffff9102bb63b0c0     0 82413      2 0x00000080
Apr 22 18:52:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: ll_ost_io03_002 D ffff9102bb639040     0 82414      2 0x00000080
Apr 22 18:52:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa921a849>] ? ___slab_alloc+0x209/0x4f0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: ll_ost02_003    S ffff9102bb63d140     0 82415      2 0x00000080
Apr 22 18:52:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: ll_ost01_003    S ffff9102bb63c100     0 82416      2 0x00000080
Apr 22 18:52:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: ll_ost_seq00_00 S ffff9102bb63a080     0 82417      2 0x00000080
Apr 22 18:52:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: ll_ost_seq00_00 S ffff9102bcf630c0     0 82418      2 0x00000080
Apr 22 18:52:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: ll_ost_seq01_00 S ffff9102bcf65140     0 82419      2 0x00000080
Apr 22 18:52:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: ll_ost_seq01_00 S ffff9102bcf66180     0 82420      2 0x00000080
Apr 22 18:52:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: ll_ost_seq02_00 S ffff9102bcf62080     0 82421      2 0x00000080
Apr 22 18:52:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: ll_ost_seq02_00 S ffff9102bcf64100     0 82422      2 0x00000080
Apr 22 18:52:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: ll_ost_seq03_00 S ffff9112ba5ee180     0 82423      2 0x00000080
Apr 22 18:52:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_ost_seq03_00 S ffff9112bc620000     0 82424      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_ost02_004    S ffff9112bc622080     0 82425      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1209d70>] ldlm_completion_ast+0x430/0x890 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1209720>] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc11f0dcf>] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1207ef2>] ? ldlm_interval_alloc+0xa2/0x260 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc120ac8c>] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc18016ad>] ofd_destroy_by_fid+0x1dd/0x4a0 [ofd]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_ost00_003    S ffff9112bc625140     0 82426      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_ost01_004    S ffff9112b8c430c0     0 82427      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_ost_out00_00 S ffff9112b8c46180     0 82428      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_ost_out00_00 S ffff9112af38b0c0     0 82429      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_ost_out01_00 S ffff9112af389040     0 82430      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_ost_out01_00 S ffff9112af388000     0 82431      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_ost_out02_00 S ffff9112b1fce180     0 82432      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_ost_out02_00 S ffff91129a791040     0 82433      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_ost_out03_00 S ffff91129a795140     0 82434      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_ost_out03_00 S ffff91129a7930c0     0 82435      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ll_evictor      S ffff91129a792080     0 82436      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc1251c5e>] ping_evictor_main+0x4be/0x700 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffc12517a0>] ? ptlrpc_del_timeout_client+0x190/0x190 [ptlrpc]
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:20 fir-io2-s2 kernel: ldiskfs-rsv-con S ffff91129a794100     0 82440      2 0x00000080
Apr 22 18:52:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:20 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: ll_ost01_005    S ffff9112bae10000     0 82441      2 0x00000080
Apr 22 18:52:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: kmmpd-md11      D ffff9112bae14100     0 82442      2 0x00000080
Apr 22 18:52:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa958d330>] ? md_handle_request+0xd0/0x150
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9278f0a>] __wait_on_buffer+0x2a/0x30
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc15d939d>] write_mmp_block+0x11d/0x160 [ldiskfs]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc15d95f8>] kmmpd+0x1a8/0x430 [ldiskfs]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc15d9450>] ? __dump_mmp_msg+0x70/0x70 [ldiskfs]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: jbd2/md11-8     D ffff9112bae12080     0 82443      2 0x00000080
Apr 22 18:52:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc09e32bc>] jbd2_journal_commit_transaction+0x23c/0x19b0 [jbd2]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa976778f>] ? __schedule+0x3ff/0x890
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90aa0de>] ? try_to_del_timer_sync+0x5e/0x90
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc09e9e89>] kjournald2+0xc9/0x260 [jbd2]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc09e9dc0>] ? commit_timeout+0x10/0x10 [jbd2]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: ldiskfs-rsv-con S ffff9112bae16180     0 82456      2 0x00000080
Apr 22 18:52:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: kmmpd-md7       D ffff9112bae15140     0 82457      2 0x00000080
Apr 22 18:52:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa958d330>] ? md_handle_request+0xd0/0x150
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9278f0a>] __wait_on_buffer+0x2a/0x30
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc15d939d>] write_mmp_block+0x11d/0x160 [ldiskfs]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc15d95f8>] kmmpd+0x1a8/0x430 [ldiskfs]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc15d9450>] ? __dump_mmp_msg+0x70/0x70 [ldiskfs]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: jbd2/md7-8      D ffff9112a5608000     0 82458      2 0x00000080
Apr 22 18:52:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa91581ef>] ? delayacct_end+0x8f/0xb0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9278f0a>] __wait_on_buffer+0x2a/0x30
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc09e3f02>] jbd2_journal_commit_transaction+0xe82/0x19b0 [jbd2]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc09e9e89>] kjournald2+0xc9/0x260 [jbd2]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc09e9dc0>] ? commit_timeout+0x10/0x10 [jbd2]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: ldiskfs-rsv-con S ffff91129971e180     0 82459      2 0x00000080
Apr 22 18:52:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90ba32e>] rescuer_thread+0x31e/0x3c0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90ba010>] ? process_one_work+0x440/0x440
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: ll_ost03_003    S ffff91129971a080     0 82477      2 0x00000080
Apr 22 18:52:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: ll_ost00_004    S ffff91129971b0c0     0 82478      2 0x00000080
Apr 22 18:52:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: ll_ost03_004    S ffff911299719040     0 82479      2 0x00000080
Apr 22 18:52:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: inconsistency_v S ffff91129971d140     0 82480      2 0x00000080
Apr 22 18:52:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc180aa00>] ofd_inconsistency_verification_main+0x960/0xee0 [ofd]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffc180a0a0>] ? ofd_cb_soft_sync+0x260/0x260 [ofd]
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:21 fir-io2-s2 kernel: lquota_wb_fir-O D ffff90e2b6ee1040     0 82525      2 0x00000080
Apr 22 18:52:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa937c8b4>] ? __radix_tree_lookup+0x84/0xf0
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa927882d>] ? __brelse+0x3d/0x50
Apr 22 18:52:21 fir-io2-s2 kernel: [<ffffffffa9278b1a>] ? bh_lru_install+0x18a/0x1e0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc15adda7>] ? ldiskfs_bread+0x27/0xc0 [ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16dbda8>] iam_txn_add+0x28/0x60 [osd_ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16debff>] iam_it_rec_delete+0x2f/0x4e0 [osd_ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa927882d>] ? __brelse+0x3d/0x50
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16df644>] iam_delete+0x64/0xe0 [osd_ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16c8da7>] osd_index_iam_delete+0x227/0x6e0 [osd_ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16be625>] ? osd_index_iam_lookup+0x275/0x410 [osd_ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16e02a0>] ? iam_lfix_split+0x140/0x140 [osd_ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1661dc4>] lquota_disk_write+0x1b4/0x880 [lquota]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1669e08>] qsd_update_index+0x2f8/0x580 [lquota]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16713f4>] qsd_upd_thread+0x424/0xe30 [lquota]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1670fd0>] ? qsd_upd_add+0xf0/0xf0 [lquota]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: inconsistency_v S ffff90e2b6ee5140     0 82535      2 0x00000080
Apr 22 18:52:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc180aa00>] ofd_inconsistency_verification_main+0x960/0xee0 [ofd]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc180a0a0>] ? ofd_cb_soft_sync+0x260/0x260 [ofd]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: lquota_wb_fir-O D ffff91029c525140     0 82561      2 0x00000080
Apr 22 18:52:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16f85d4>] ? dynlock_unlock+0x194/0x1e0 [osd_ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90e060c>] ? dequeue_entity+0x11c/0x5e0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1669d0a>] qsd_update_index+0x1fa/0x580 [lquota]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16713f4>] qsd_upd_thread+0x424/0xe30 [lquota]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1670fd0>] ? qsd_upd_add+0xf0/0xf0 [lquota]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: ll_ost02_005    D ffff910290bda080     0 82568      2 0x00000080
Apr 22 18:52:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: ll_ost02_006    D ffff910290bd9040     0 82583      2 0x00000080
Apr 22 18:52:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: ll_ost03_005    S ffff9102b8f44100     0 82584      2 0x00000080
Apr 22 18:52:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: inconsistency_v S ffff9102b8f46180     0 82585      2 0x00000080
Apr 22 18:52:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc180aa00>] ofd_inconsistency_verification_main+0x960/0xee0 [ofd]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc180a0a0>] ? ofd_cb_soft_sync+0x260/0x260 [ofd]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: lquota_wb_fir-O S ffff9112b2328000     0 82611      2 0x00000080
Apr 22 18:52:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1670abe>] ? qsd_job_pending+0x7e/0x1b0 [lquota]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1671ab6>] qsd_upd_thread+0xae6/0xe30 [lquota]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc1670fd0>] ? qsd_upd_add+0xf0/0xf0 [lquota]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: inconsistency_v S ffff90e2a8f11040     0 82637      2 0x00000080
Apr 22 18:52:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc180aa00>] ofd_inconsistency_verification_main+0x960/0xee0 [ofd]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffc180a0a0>] ? ofd_cb_soft_sync+0x260/0x260 [ofd]
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: lquota_wb_fir-O D ffff90fc96ccc100     0 82663      2 0x00000080
Apr 22 18:52:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc16f85d4>] ? dynlock_unlock+0x194/0x1e0 [osd_ldiskfs]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90e060c>] ? dequeue_entity+0x11c/0x5e0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1669d0a>] qsd_update_index+0x1fa/0x580 [lquota]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc16713f4>] qsd_upd_thread+0x424/0xe30 [lquota]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1670fd0>] ? qsd_upd_add+0xf0/0xf0 [lquota]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: inconsistency_v S ffff90fc96cc8000     0 82675      2 0x00000080
Apr 22 18:52:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc180aa00>] ofd_inconsistency_verification_main+0x960/0xee0 [ofd]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc180a0a0>] ? ofd_cb_soft_sync+0x260/0x260 [ofd]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: lquota_wb_fir-O S ffff90e2b58a4100     0 82701      2 0x00000080
Apr 22 18:52:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1670abe>] ? qsd_job_pending+0x7e/0x1b0 [lquota]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1671ab6>] qsd_upd_thread+0xae6/0xe30 [lquota]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1670fd0>] ? qsd_upd_add+0xf0/0xf0 [lquota]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: ll_ost01_006    S ffff90e2b58a1040     0 82711      2 0x00000080
Apr 22 18:52:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: inconsistency_v S ffff90e2b58a5140     0 82712      2 0x00000080
Apr 22 18:52:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc180aa00>] ofd_inconsistency_verification_main+0x960/0xee0 [ofd]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc180a0a0>] ? ofd_cb_soft_sync+0x260/0x260 [ofd]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: lquota_wb_fir-O D ffff90e2b7580000     0 82738      2 0x00000080
Apr 22 18:52:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc15adda7>] ? ldiskfs_bread+0x27/0xc0 [ldiskfs]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc16dbda8>] iam_txn_add+0x28/0x60 [osd_ldiskfs]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc16debff>] iam_it_rec_delete+0x2f/0x4e0 [osd_ldiskfs]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc16df644>] iam_delete+0x64/0xe0 [osd_ldiskfs]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc16c8da7>] osd_index_iam_delete+0x227/0x6e0 [osd_ldiskfs]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc16be625>] ? osd_index_iam_lookup+0x275/0x410 [osd_ldiskfs]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc16e02a0>] ? iam_lfix_split+0x140/0x140 [osd_ldiskfs]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1661dc4>] lquota_disk_write+0x1b4/0x880 [lquota]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1669e08>] qsd_update_index+0x2f8/0x580 [lquota]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc16713f4>] qsd_upd_thread+0x424/0xe30 [lquota]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1670fd0>] ? qsd_upd_add+0xf0/0xf0 [lquota]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: ll_ost00_005    S ffff90e2b7585140     0 82758      2 0x00000080
Apr 22 18:52:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: ll_ost00_006    S ffff910298a51040     0 82791      2 0x00000080
Apr 22 18:52:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: ll_ost01_007    S ffff91027969d140     0 82867      2 0x00000080
Apr 22 18:52:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: ll_ost00_007    S ffff90e2bc7a5140     0 82940      2 0x00000080
Apr 22 18:52:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: ll_ost01_008    D ffff90d3e95e0000     0 82943      2 0x00000080
Apr 22 18:52:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:23 fir-io2-s2 kernel: ll_ost00_008    S ffff9112bca48000     0 83060      2 0x00000080
Apr 22 18:52:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:23 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: ll_ost01_009    S ffff90e2b3c22080     0 83727      2 0x00000080
Apr 22 18:52:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: ll_ost01_010    S ffff90e2b54f4100     0 83728      2 0x00000080
Apr 22 18:52:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: ll_ost01_011    S ffff90e2b54f2080     0 83729      2 0x00000080
Apr 22 18:52:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: ll_ost01_012    R  running task        0 83730      2 0x00000080
Apr 22 18:52:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124f00c>] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] ? kthread+0xd1/0xe0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ? ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: ll_ost01_013    S ffff90e2b54f5140     0 83731      2 0x00000080
Apr 22 18:52:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: ll_ost01_014    S ffff90e2b54f0000     0 83732      2 0x00000080
Apr 22 18:52:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1209d70>] ldlm_completion_ast+0x430/0x890 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1209720>] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc11f0dcf>] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1207ef2>] ? ldlm_interval_alloc+0xa2/0x260 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc120ac8c>] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc18016ad>] ofd_destroy_by_fid+0x1dd/0x4a0 [ofd]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: ll_ost00_009    S ffff90e2b54f30c0     0 83733      2 0x00000080
Apr 22 18:52:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: ll_ost01_015    S ffff91029adf30c0     0 83734      2 0x00000080
Apr 22 18:52:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: ll_ost00_010    S ffff90fef7375140     0 83735      2 0x00000080
Apr 22 18:52:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: ll_ost01_016    S ffff90fef73730c0     0 83736      2 0x00000080
Apr 22 18:52:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: ll_ost00_011    S ffff90fef7376180     0 83737      2 0x00000080
Apr 22 18:52:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:24 fir-io2-s2 kernel: ll_ost01_017    D ffff90fef7371040     0 83738      2 0x00000080
Apr 22 18:52:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa93a0160>] ? __percpu_counter_sum+0x70/0x80
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 18:52:24 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: ll_ost00_012    S ffff90fef7374100     0 83739      2 0x00000080
Apr 22 18:52:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: ll_ost01_018    S ffff90fef7370000     0 83740      2 0x00000080
Apr 22 18:52:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: ll_ost01_019    S ffff90fef7372080     0 83741      2 0x00000080
Apr 22 18:52:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: ll_ost01_020    S ffff9102bacc5140     0 83742      2 0x00000080
Apr 22 18:52:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc07dbffc>] ? dm_old_request_fn+0xcc/0x210 [dm_mod]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9381153>] ? number.isra.2+0x323/0x360
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa93824d9>] ? pointer.isra.19+0x1c9/0x4d0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc0d12632>] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc15f3eda>] ? ldiskfs_xattr_trusted_get+0x2a/0x30 [ldiskfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9269092>] ? generic_getxattr+0x52/0x70
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc0d12bd4>] ? kiblnd_launch_tx+0x184/0xc20 [ko2iblnd]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc0d139c7>] ? kiblnd_send+0x357/0xa20 [ko2iblnd]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9381153>] ? number.isra.2+0x323/0x360
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1056fd5>] ? lu_object_alloc+0xe5/0x320 [obdclass]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc10572e6>] ? lu_object_find_at+0x76/0x280 [obdclass]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1057506>] ? lu_object_find+0x16/0x20 [obdclass]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1805185>] ? ofd_object_find+0x35/0x100 [ofd]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc18146f1>] ? ofd_lvbo_init+0x391/0x848 [ofd]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa902a59e>] ? __switch_to+0xce/0x580
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90e060c>] ? dequeue_entity+0x11c/0x5e0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1219600>] ? ldlm_handle_enqueue0+0x970/0x15a0 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90d9ed5>] ? sched_clock_cpu+0x85/0xc0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa91656c5>] ? tracing_is_on+0x15/0x30
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa91717bd>] ? probe_sched_switch+0x2d/0xa0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc12a00b2>] ? tgt_enqueue+0x62/0x210 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc12a710a>] ? tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] ? schedule+0x29/0x70
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124b6db>] ? ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124ed63>] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] ? kthread+0xd1/0xe0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ? ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: ll_ost01_021    D ffff9102bacc6180     0 83743      2 0x00000080
Apr 22 18:52:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: ll_ost01_022    D ffff9102bacc1040     0 83744      2 0x00000080
Apr 22 18:52:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa927882d>] ? __brelse+0x3d/0x50
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc16d8f00>] ? osd_oi_insert+0x4d0/0x520 [osd_ldiskfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc16c7edf>] ? osd_destroy+0x1af/0x760 [osd_ldiskfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc16c723d>] ? osd_ref_del+0x1ad/0x6a0 [osd_ldiskfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1809455>] ofd_destroy+0x2c5/0x960 [ofd]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: ll_ost00_013    D ffff9102bacc2080     0 83745      2 0x00000080
Apr 22 18:52:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: ll_ost03_006    S ffff9102bacc30c0     0 83746      2 0x00000080
Apr 22 18:52:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: ll_ost01_023    S ffff9102bc661040     0 83747      2 0x00000080
Apr 22 18:52:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: ll_ost01_024    S ffff9102bc666180     0 83748      2 0x00000080
Apr 22 18:52:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: ll_ost01_025    S ffff9102bc665140     0 83749      2 0x00000080
Apr 22 18:52:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: ll_ost00_014    S ffff9102bc662080     0 83750      2 0x00000080
Apr 22 18:52:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: ll_ost01_026    S ffff90f369fdb0c0     0 83751      2 0x00000080
Apr 22 18:52:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: ll_ost01_027    S ffff9112b03730c0     0 83752      2 0x00000080
Apr 22 18:52:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: ll_ost03_007    S ffff9112b0374100     0 83753      2 0x00000080
Apr 22 18:52:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: ll_ost01_028    S ffff9112b0372080     0 83754      2 0x00000080
Apr 22 18:52:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:26 fir-io2-s2 kernel: ll_ost01_029    S ffff90e2bbb5e180     0 83755      2 0x00000080
Apr 22 18:52:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc07dbffc>] ? dm_old_request_fn+0xcc/0x210 [dm_mod]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa936786e>] ? deadline_add_request+0x6e/0x90
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9381153>] ? number.isra.2+0x323/0x360
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9765d71>] ? bit_wait_io+0x11/0x50
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc15a534f>] ? ldiskfs_dx_find_entry+0xef/0x200 [ldiskfs]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9278f0a>] ? __wait_on_buffer+0x2a/0x30
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc15ae346>] ? ldiskfs_iget+0x96/0xc50 [ldiskfs]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc16c4b16>] ? osd_iget+0x26/0x2d0 [osd_ldiskfs]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc0d12632>] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc16d8989>] ? osd_oi_lookup+0x139/0x1e0 [osd_ldiskfs]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc16d3ea5>] ? osd_fid_lookup+0x4a5/0x1c70 [osd_ldiskfs]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc0d12bd4>] ? kiblnd_launch_tx+0x184/0xc20 [ko2iblnd]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc0d139c7>] ? kiblnd_send+0x357/0xa20 [ko2iblnd]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90de6bc>] ? set_next_entity+0x3c/0xe0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc10572e6>] ? lu_object_find_at+0x76/0x280 [obdclass]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa921a849>] ? ___slab_alloc+0x209/0x4f0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc0c83f65>] ? lnet_md_alloc.isra.5+0x145/0x300 [lnet]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90e060c>] ? dequeue_entity+0x11c/0x5e0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90d9ed5>] ? sched_clock_cpu+0x85/0xc0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa91656c5>] ? tracing_is_on+0x15/0x30
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa91717bd>] ? probe_sched_switch+0x2d/0xa0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] ? schedule+0x29/0x70
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc1248805>] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124ed63>] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] ? kthread+0xd1/0xe0
Apr 22 18:52:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ? ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost00_015    S ffff90e2bbb58000     0 83756      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost01_030    S ffff90e2bbb5b0c0     0 83757      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost00_016    S ffff90e2bbb5c100     0 83758      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost01_031    S ffff90e2b3fc8000     0 83759      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost00_017    S ffff90e2b3fc9040     0 83760      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost01_032    S ffff90e2b3fce180     0 83761      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9381153>] ? number.isra.2+0x323/0x360
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc15a534f>] ? ldiskfs_dx_find_entry+0xef/0x200 [ldiskfs]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc15a592b>] ? __ldiskfs_find_entry+0x4cb/0x580 [ldiskfs]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9382c21>] ? vsnprintf+0x441/0x6a0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc15ac9fd>] ? ldiskfs_get_inode_loc+0x1d/0x20 [ldiskfs]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc15b61ae>] ? ldiskfs_xattr_ibody_get+0x4e/0x1a0 [ldiskfs]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc15b636d>] ? ldiskfs_xattr_get+0x6d/0x2f0 [ldiskfs]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc0d12632>] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc0d12bd4>] ? kiblnd_launch_tx+0x184/0xc20 [ko2iblnd]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc0d139c7>] ? kiblnd_send+0x357/0xa20 [ko2iblnd]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9381153>] ? number.isra.2+0x323/0x360
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc0c890e4>] ? lnet_ni_send+0x44/0xd0 [lnet]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc0c90d92>] ? lnet_send+0x82/0x1c0 [lnet]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1057506>] ? lu_object_find+0x16/0x20 [obdclass]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1805185>] ? ofd_object_find+0x35/0x100 [ofd]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc18146f1>] ? ofd_lvbo_init+0x391/0x848 [ofd]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc123973b>] ? ptlrpc_send_reply+0x29b/0x840 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90e060c>] ? dequeue_entity+0x11c/0x5e0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d9ed5>] ? sched_clock_cpu+0x85/0xc0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa91656c5>] ? tracing_is_on+0x15/0x30
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa91691bd>] ? tracing_record_cmdline+0x1d/0x120
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa91717bd>] ? probe_sched_switch+0x2d/0xa0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] ? schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124f00c>] ? ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] ? kthread+0xd1/0xe0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ? ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost00_018    S ffff90e2b3fca080     0 83762      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost00_019    S ffff90e2b3fcd140     0 83763      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost01_033    S ffff90e2b3fcc100     0 83764      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost00_020    S ffff90d3e957a080     0 83765      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost01_034    S ffff90e2a936b0c0     0 83766      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost00_021    S ffff90e2a936c100     0 83767      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:27 fir-io2-s2 kernel: ll_ost01_035    S ffff90e2ae22e180     0 83768      2 0x00000080
Apr 22 18:52:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: ll_ost01_036    S ffff90e2ae22c100     0 83769      2 0x00000080
Apr 22 18:52:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: ll_ost03_008    S ffff90e2ae22d140     0 83770      2 0x00000080
Apr 22 18:52:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: ll_ost00_022    S ffff90e2a8b430c0     0 83771      2 0x00000080
Apr 22 18:52:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: ll_ost01_037    D ffff90e2a8b40000     0 83772      2 0x00000080
Apr 22 18:52:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc0c5aff2>] ? fld_local_lookup+0x62/0x270 [fld]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1805f17>] ofd_precreate_objects+0xa57/0x1ab0 [ofd]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc17f8794>] ofd_create_hdl+0x474/0x20e0 [ofd]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc123f4b7>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: ll_ost03_009    S ffff90e2a8b44100     0 83773      2 0x00000080
Apr 22 18:52:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: ll_ost01_038    S ffff90e2a8b46180     0 83774      2 0x00000080
Apr 22 18:52:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: ll_ost00_023    D ffff90e2a8b45140     0 83775      2 0x00000080
Apr 22 18:52:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: ll_ost00_024    D ffff90e2a8b41040     0 83776      2 0x00000080
Apr 22 18:52:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90d6705>] ? wake_up_process+0x15/0x20
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa91581ef>] ? delayacct_end+0x8f/0xb0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9158314>] ? __delayacct_blkio_end+0x34/0x60
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9767327>] io_schedule_timeout+0xe7/0x130
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa97658ae>] ? __wait_on_bit+0x7e/0x90
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9765a01>] ? out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9381153>] ? number.isra.2+0x323/0x360
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa93824d9>] ? pointer.isra.19+0x1c9/0x4d0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc0d12632>] ? kiblnd_check_sends_locked+0xa72/0xe40 [ko2iblnd]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc0d12bd4>] ? kiblnd_launch_tx+0x184/0xc20 [ko2iblnd]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc0d139c7>] ? kiblnd_send+0x357/0xa20 [ko2iblnd]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9381153>] ? number.isra.2+0x323/0x360
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1056fd5>] ? lu_object_alloc+0xe5/0x320 [obdclass]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc10572e6>] ? lu_object_find_at+0x76/0x280 [obdclass]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1057506>] ? lu_object_find+0x16/0x20 [obdclass]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1805185>] ? ofd_object_find+0x35/0x100 [ofd]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa921b581>] ? __slab_free+0x81/0x2f0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90e060c>] ? dequeue_entity+0x11c/0x5e0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90e0eee>] ? dequeue_task_fair+0x41e/0x660
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc12a00b2>] ? tgt_enqueue+0x62/0x210 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc12a710a>] ? tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] ? schedule+0x29/0x70
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1248805>] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124ed63>] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] ? kthread+0xd1/0xe0
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ? ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:28 fir-io2-s2 kernel: ll_ost03_010    S ffff90d2d7070000     0 83777      2 0x00000080
Apr 22 18:52:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:28 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost01_039    S ffff90d2d7071040     0 83778      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost03_011    S ffff90d2d7075140     0 83779      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost01_040    S ffff90d2d7074100     0 83780      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost01_041    S ffff90d2d7076180     0 83781      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost01_042    D ffff90f2b81d9040     0 83782      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9769535>] rwsem_down_write_failed+0x225/0x3a0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9386c47>] call_rwsem_down_write_failed+0x17/0x30
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9766f4d>] down_write+0x2d/0x3d
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc16ba97c>] osd_write_lock+0x5c/0xe0 [osd_ldiskfs]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1807994>] ofd_attr_set+0x84/0xb50 [ofd]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc17fa71d>] ofd_setattr_hdl+0x31d/0x8f0 [ofd]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost00_025    S ffff90f2b81dc100     0 83783      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost01_043    S ffff90f2b68e0000     0 83784      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost00_026    S ffff90f2b68e2080     0 83785      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost01_044    S ffff90f2b68e1040     0 83786      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost01_045    S ffff90f2b68e4100     0 83787      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost00_027    S ffff90f2b68e30c0     0 83788      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost00_028    D ffff90f2b68e5140     0 83789      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost01_046    S ffff90f2b92c9040     0 83790      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:29 fir-io2-s2 kernel: ll_ost01_047    S ffff90f2b92cc100     0 83791      2 0x00000080
Apr 22 18:52:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: ll_ost01_048    S ffff90f2bb6a0000     0 83792      2 0x00000080
Apr 22 18:52:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: ll_ost00_029    S ffff90f2bb6a2080     0 83793      2 0x00000080
Apr 22 18:52:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: ll_ost00_030    S ffff90f2b7a61040     0 83794      2 0x00000080
Apr 22 18:52:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: ll_ost00_031    S ffff90f2bb5e0000     0 83795      2 0x00000080
Apr 22 18:52:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: ll_ost01_049    S ffff90f2b8bba080     0 83796      2 0x00000080
Apr 22 18:52:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: ll_ost_io01_003 D ffff90f2b9500000     0 84712      2 0x00000080
Apr 22 18:52:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: ll_ost_io02_003 D ffff90f2b9501040     0 84713      2 0x00000080
Apr 22 18:52:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:30 fir-io2-s2 kernel: ll_ost_io00_003 D ffff90f2b9505140     0 84714      2 0x00000080
Apr 22 18:52:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:30 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: ll_ost_io03_003 D ffff90f2b95030c0     0 84715      2 0x00000080
Apr 22 18:52:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: ll_ost02_007    D ffff90f2b9504100     0 84716      2 0x00000080
Apr 22 18:52:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: ll_ost02_008    D ffff90e2b99d0000     0 84717      2 0x00000080
Apr 22 18:52:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc0beebde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1259538>] ? at_measured+0x108/0x360 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc15b0dc0>] ldiskfs_reserve_inode_write+0x70/0xa0 [ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ? ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc15b0e43>] ldiskfs_mark_inode_dirty+0x53/0x210 [ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc16c722d>] osd_ref_del+0x19d/0x6a0 [osd_ldiskfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1809699>] ofd_destroy+0x509/0x960 [ofd]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: ll_ost02_009    D ffff90e2b99d5140     0 84718      2 0x00000080
Apr 22 18:52:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: ll_ost02_010    S ffff90e2b1b86180     0 84719      2 0x00000080
Apr 22 18:52:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: ll_ost02_011    D ffff91028e6c1040     0 84720      2 0x00000080
Apr 22 18:52:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc0beebde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc1259538>] ? at_measured+0x108/0x360 [ptlrpc]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 18:52:31 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc15b0dc0>] ldiskfs_reserve_inode_write+0x70/0xa0 [ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ? ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc15b0e43>] ldiskfs_mark_inode_dirty+0x53/0x210 [ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc16c722d>] osd_ref_del+0x19d/0x6a0 [osd_ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1809699>] ofd_destroy+0x509/0x960 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: ll_ost03_012    D ffff91028e6c2080     0 84721      2 0x00000080
Apr 22 18:52:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: ll_ost02_012    D ffff90feacfea080     0 84722      2 0x00000080
Apr 22 18:52:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc15a53e6>] ? ldiskfs_dx_find_entry+0x186/0x200 [ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9765b71>] out_of_line_wait_on_bit_lock+0x81/0xb0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9278f42>] __lock_buffer+0x32/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc09e1ea2>] do_get_write_access+0x432/0x4d0 [jbd2]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc180642d>] ofd_precreate_objects+0xf6d/0x1ab0 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc17f8794>] ofd_create_hdl+0x474/0x20e0 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc123f4b7>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: ll_ost03_013    S ffff90feacfe9040     0 84723      2 0x00000080
Apr 22 18:52:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: ll_ost02_013    D ffff90feacfee180     0 84724      2 0x00000080
Apr 22 18:52:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: ll_ost02_014    D ffff90feacfe8000     0 84725      2 0x00000080
Apr 22 18:52:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:32 fir-io2-s2 kernel: ll_ost02_015    D ffff90feacfeb0c0     0 84726      2 0x00000080
Apr 22 18:52:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: ll_ost02_016    D ffff90feacfed140     0 84727      2 0x00000080
Apr 22 18:52:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: ll_ost03_014    S ffff90feacfec100     0 84728      2 0x00000080
Apr 22 18:52:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: ll_ost02_017    D ffff910275b1c100     0 84729      2 0x00000080
Apr 22 18:52:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: ll_ost02_018    S ffff910275b1e180     0 84730      2 0x00000080
Apr 22 18:52:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: ll_ost01_050    D ffff910275b1a080     0 84731      2 0x00000080
Apr 22 18:52:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90e3ce4>] ? find_busiest_group+0x144/0x990
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9381153>] ? number.isra.2+0x323/0x360
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc15caac3>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc15caac3>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa92aed8a>] dqget+0x3fa/0x450
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa92afb54>] dquot_get_dqblk+0x14/0x1f0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc16f8f55>] osd_acct_index_lookup+0x235/0x480 [osd_ldiskfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc165e80d>] lquotactl_slv+0x27d/0x9d0 [lquota]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc17f1a3c>] ofd_quotactl+0x13c/0x380 [ofd]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: ll_ost01_051    D ffff910275b1b0c0     0 84732      2 0x00000080
Apr 22 18:52:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:33 fir-io2-s2 kernel: ll_ost02_019    D ffff910275b1d140     0 84733      2 0x00000080
Apr 22 18:52:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: ll_ost02_020    D ffff910275b19040     0 84734      2 0x00000080
Apr 22 18:52:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc15b0dc0>] ldiskfs_reserve_inode_write+0x70/0xa0 [ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ? ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc15b0e43>] ldiskfs_mark_inode_dirty+0x53/0x210 [ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc16c68f1>] osd_attr_set+0x3f1/0xb90 [osd_ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1807ea3>] ofd_attr_set+0x593/0xb50 [ofd]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc17fa71d>] ofd_setattr_hdl+0x31d/0x8f0 [ofd]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: ll_ost01_052    S ffff910275b18000     0 84735      2 0x00000080
Apr 22 18:52:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: ll_ost03_015    S ffff90f2b893c100     0 84736      2 0x00000080
Apr 22 18:52:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: ll_ost01_053    S ffff90f2b893b0c0     0 84737      2 0x00000080
Apr 22 18:52:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: ll_ost02_021    D ffff90f2b893a080     0 84738      2 0x00000080
Apr 22 18:52:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1805f17>] ofd_precreate_objects+0xa57/0x1ab0 [ofd]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc17f8794>] ofd_create_hdl+0x474/0x20e0 [ofd]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc123f4b7>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: ll_ost01_054    D ffff90f2b8939040     0 84739      2 0x00000080
Apr 22 18:52:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: ll_ost03_016    S ffff90f2b8ac8000     0 84740      2 0x00000080
Apr 22 18:52:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa976a175>] ? _raw_spin_unlock_irqrestore+0x15/0x20
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: ll_ost02_022    S ffff90e36983b0c0     0 84741      2 0x00000080
Apr 22 18:52:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:34 fir-io2-s2 kernel: ll_ost01_055    S ffff90e369839040     0 84742      2 0x00000080
Apr 22 18:52:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:34 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: ll_ost02_023    D ffff9100d6478000     0 84743      2 0x00000080
Apr 22 18:52:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: ll_ost02_024    D ffff9100d6479040     0 84744      2 0x00000080
Apr 22 18:52:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: ll_ost02_025    D ffff9100d647a080     0 84745      2 0x00000080
Apr 22 18:52:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: ll_ost01_056    S ffff9100d647b0c0     0 84746      2 0x00000080
Apr 22 18:52:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: ll_ost01_057    S ffff9100d647c100     0 84747      2 0x00000080
Apr 22 18:52:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: ll_ost00_032    S ffff9100d647d140     0 84748      2 0x00000080
Apr 22 18:52:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:35 fir-io2-s2 kernel: ll_ost02_026    D ffff9100d647e180     0 84749      2 0x00000080
Apr 22 18:52:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc09e22d1>] ? jbd2_journal_dirty_metadata+0xd1/0x260 [jbd2]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc12977f5>] tgt_last_rcvd_update+0x295/0xc90 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16d8f00>] ? osd_oi_insert+0x4d0/0x520 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16c7edf>] ? osd_destroy+0x1af/0x760 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc16c723d>] ? osd_ref_del+0x1ad/0x6a0 [osd_ldiskfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1809455>] ofd_destroy+0x2c5/0x960 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:35 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: ll_ost03_017    D ffff9100cfcb8000     0 84750      2 0x00000080
Apr 22 18:52:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa93824d9>] ? pointer.isra.19+0x1c9/0x4d0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1298909>] tgt_client_data_update+0x3a9/0x5e0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: ll_ost01_058    S ffff9100cfcb9040     0 84751      2 0x00000080
Apr 22 18:52:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: ll_ost03_018    S ffff9100cfcba080     0 84752      2 0x00000080
Apr 22 18:52:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: ll_ost01_059    D ffff9100cfcbb0c0     0 84753      2 0x00000080
Apr 22 18:52:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: ll_ost02_027    D ffff9100cfcbc100     0 84754      2 0x00000080
Apr 22 18:52:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: ll_ost00_033    S ffff9100cfcbd140     0 84755      2 0x00000080
Apr 22 18:52:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: ll_ost00_034    S ffff9100cfcbe180     0 84756      2 0x00000080
Apr 22 18:52:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: ll_ost02_028    D ffff90febc060000     0 84757      2 0x00000080
Apr 22 18:52:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc09e22d1>] ? jbd2_journal_dirty_metadata+0xd1/0x260 [jbd2]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc12977f5>] tgt_last_rcvd_update+0x295/0xc90 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc16d8f00>] ? osd_oi_insert+0x4d0/0x520 [osd_ldiskfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc16c7edf>] ? osd_destroy+0x1af/0x760 [osd_ldiskfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc16c723d>] ? osd_ref_del+0x1ad/0x6a0 [osd_ldiskfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1809455>] ofd_destroy+0x2c5/0x960 [ofd]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:36 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost01_060    S ffff90febc061040     0 84758      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost03_019    S ffff90febc062080     0 84759      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa976a175>] ? _raw_spin_unlock_irqrestore+0x15/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost01_061    S ffff90febc0630c0     0 84760      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost03_020    S ffff90febc064100     0 84761      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost01_062    S ffff90febc065140     0 84762      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost02_029    D ffff90febc066180     0 84763      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost00_035    S ffff9100d0088000     0 84764      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost01_063    S ffff9100d0089040     0 84765      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost00_036    S ffff9100d008a080     0 84766      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost02_030    D ffff9100d008b0c0     0 84767      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost00_037    S ffff9100d008c100     0 84768      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost03_021    S ffff9100d008d140     0 84769      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:37 fir-io2-s2 kernel: ll_ost01_064    S ffff9100d008e180     0 84770      2 0x00000080
Apr 22 18:52:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: ll_ost01_065    S ffff9100cb0c0000     0 84771      2 0x00000080
Apr 22 18:52:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: ll_ost00_038    S ffff9100cb0c1040     0 84772      2 0x00000080
Apr 22 18:52:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: ll_ost01_066    S ffff9100cb0c2080     0 84773      2 0x00000080
Apr 22 18:52:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: ll_ost01_067    D ffff9100cb0c30c0     0 84774      2 0x00000080
Apr 22 18:52:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: ll_ost02_031    D ffff9100cb0c4100     0 84775      2 0x00000080
Apr 22 18:52:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: ll_ost00_039    D ffff9100cb0c5140     0 84776      2 0x00000080
Apr 22 18:52:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: ll_ost01_068    S ffff9100cb0c6180     0 84777      2 0x00000080
Apr 22 18:52:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: ldlm_cn00_003   S ffff9100cb5f8000     0 84778      2 0x00000080
Apr 22 18:52:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: ll_ost01_069    D ffff9100cb5f9040     0 84779      2 0x00000080
Apr 22 18:52:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc0d139c7>] ? kiblnd_send+0x357/0xa20 [ko2iblnd]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9381153>] ? number.isra.2+0x323/0x360
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc15caac3>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc15caac3>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa92aed8a>] dqget+0x3fa/0x450
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa92afb54>] dquot_get_dqblk+0x14/0x1f0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc16f8f55>] osd_acct_index_lookup+0x235/0x480 [osd_ldiskfs]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc165e80d>] lquotactl_slv+0x27d/0x9d0 [lquota]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc17f1a3c>] ofd_quotactl+0x13c/0x380 [ofd]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: ll_ost00_040    S ffff9100cb5fa080     0 84780      2 0x00000080
Apr 22 18:52:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: ll_ost03_022    S ffff9100cb5fb0c0     0 84781      2 0x00000080
Apr 22 18:52:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: ll_ost02_032    D ffff9100cb5fc100     0 84782      2 0x00000080
Apr 22 18:52:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: ll_ost01_070    S ffff9100cb5fd140     0 84783      2 0x00000080
Apr 22 18:52:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: ll_ost02_033    D ffff9100cb5fe180     0 84784      2 0x00000080
Apr 22 18:52:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: ll_ost00_041    S ffff9100c7c48000     0 84785      2 0x00000080
Apr 22 18:52:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: ll_ost01_071    D ffff9100c7c49040     0 84786      2 0x00000080
Apr 22 18:52:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: ll_ost02_034    D ffff9100c7c4a080     0 84787      2 0x00000080
Apr 22 18:52:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: ll_ost03_023    S ffff9100c7c4b0c0     0 84788      2 0x00000080
Apr 22 18:52:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: ll_ost00_042    S ffff9100c7c4c100     0 84789      2 0x00000080
Apr 22 18:52:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:39 fir-io2-s2 kernel: ll_ost02_035    D ffff9100c7c4d140     0 84790      2 0x00000080
Apr 22 18:52:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:39 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: ll_ost01_072    S ffff9100c7c4e180     0 84791      2 0x00000080
Apr 22 18:52:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: ll_ost00_043    S ffff9100c6a90000     0 84792      2 0x00000080
Apr 22 18:52:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: ll_ost02_036    D ffff9100c6a91040     0 84793      2 0x00000080
Apr 22 18:52:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: ll_ost03_024    S ffff9100c6a92080     0 84794      2 0x00000080
Apr 22 18:52:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: ll_ost02_037    D ffff9100c6a930c0     0 84795      2 0x00000080
Apr 22 18:52:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: ll_ost01_073    D ffff90ebe1549450     0 84796      2 0x00000080
Apr 22 18:52:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1218dfa>] ? ldlm_handle_enqueue0+0x16a/0x15a0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc12415a0>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc12a00b2>] ? tgt_enqueue+0x62/0x210 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc12a710a>] ? tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] ? schedule+0x29/0x70
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1248805>] ? ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124ed63>] ? ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] ? kthread+0xd1/0xe0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ? ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: ll_ost00_044    S ffff9100c6a95140     0 84797      2 0x00000080
Apr 22 18:52:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: ll_ost03_025    S ffff9100c6a96180     0 84798      2 0x00000080
Apr 22 18:52:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: ll_ost01_074    D ffff910041c48000     0 84799      2 0x00000080
Apr 22 18:52:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffa927882d>] ? __brelse+0x3d/0x50
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 18:52:40 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc16d8f00>] ? osd_oi_insert+0x4d0/0x520 [osd_ldiskfs]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc16c7edf>] ? osd_destroy+0x1af/0x760 [osd_ldiskfs]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc16c723d>] ? osd_ref_del+0x1ad/0x6a0 [osd_ldiskfs]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc1809455>] ofd_destroy+0x2c5/0x960 [ofd]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:41 fir-io2-s2 kernel: ll_ost00_045    S ffff910041c49040     0 84800      2 0x00000080
Apr 22 18:52:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:41 fir-io2-s2 kernel: ll_ost02_038    D ffff910041c4a080     0 84801      2 0x00000080
Apr 22 18:52:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:41 fir-io2-s2 kernel: ll_ost01_075    S ffff910041c4b0c0     0 84802      2 0x00000080
Apr 22 18:52:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:42 fir-io2-s2 kernel: ll_ost01_076    S ffff910041c4c100     0 84803      2 0x00000080
Apr 22 18:52:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:42 fir-io2-s2 kernel: ll_ost00_046    S ffff910041c4d140     0 84804      2 0x00000080
Apr 22 18:52:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:42 fir-io2-s2 kernel: ll_ost01_077    S ffff910041c4e180     0 84805      2 0x00000080
Apr 22 18:52:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:42 fir-io2-s2 kernel: ll_ost01_078    S ffff90fd41978000     0 84806      2 0x00000080
Apr 22 18:52:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:43 fir-io2-s2 kernel: ll_ost02_039    D ffff90fd41979040     0 84807      2 0x00000080
Apr 22 18:52:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:43 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:43 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:43 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:43 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:43 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:43 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:43 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:44 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:44 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:44 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:44 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:44 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:44 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:44 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:44 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:44 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:44 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:44 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:44 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:45 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:45 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:45 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:45 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:45 fir-io2-s2 kernel: ll_ost00_047    S ffff90fd4197a080     0 84808      2 0x00000080
Apr 22 18:52:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:45 fir-io2-s2 kernel: Lustre: fir-OST000d: Export ffff90e188bb5400 already connecting from 10.9.106.36@o2ib4
Apr 22 18:52:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:45 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:46 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:46 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:46 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:46 fir-io2-s2 kernel: ll_ost01_079    D ffff90fd4197b0c0     0 84809      2 0x00000080
Apr 22 18:52:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:46 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:46 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:46 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:46 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:47 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:47 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:47 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:47 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:47 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:47 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:47 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:47 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:47 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:48 fir-io2-s2 kernel: ll_ost00_048    S ffff90fd4197c100     0 84810      2 0x00000080
Apr 22 18:52:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:48 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:48 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:48 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:48 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:48 fir-io2-s2 kernel: ll_ost02_040    D ffff90fd4197d140     0 84811      2 0x00000080
Apr 22 18:52:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:48 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:50 fir-io2-s2 kernel: ll_ost02_041    D ffff90fd4197e180     0 84812      2 0x00000080
Apr 22 18:52:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffc0beebde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:52:50 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffa9765b71>] out_of_line_wait_on_bit_lock+0x81/0xb0
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffa9278f42>] __lock_buffer+0x32/0x40
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffc09e1ea2>] do_get_write_access+0x432/0x4d0 [jbd2]
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffc15b0dc0>] ldiskfs_reserve_inode_write+0x70/0xa0 [ldiskfs]
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ? ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 18:52:51 fir-io2-s2 kernel: [<ffffffffc15b0e43>] ldiskfs_mark_inode_dirty+0x53/0x210 [ldiskfs]
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffc16c722d>] osd_ref_del+0x19d/0x6a0 [osd_ldiskfs]
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffc1809699>] ofd_destroy+0x509/0x960 [ofd]
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:52 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:53 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:53 fir-io2-s2 kernel: ll_ost01_080    D ffff90ff994c0000     0 84813      2 0x00000080
Apr 22 18:52:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:53 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:53 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:53 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:53 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:53 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:54 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:54 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:54 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:54 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:54 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:54 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:54 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:54 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:54 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:54 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:54 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:54 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:55 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:55 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:55 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:55 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:55 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:55 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:55 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:55 fir-io2-s2 kernel: ll_ost02_042    D ffff90ff994c1040     0 84814      2 0x00000080
Apr 22 18:52:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:55 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:57 fir-io2-s2 kernel: ll_ost00_049    S ffff90ff994c2080     0 84815      2 0x00000080
Apr 22 18:52:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:57 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:52:57 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:52:57 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:52:57 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:52:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:52:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:52:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:52:58 fir-io2-s2 kernel: ll_ost01_081    D ffff90ff994c30c0     0 84816      2 0x00000080
Apr 22 18:52:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:52:58 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:52:59 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:52:59 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:52:59 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:52:59 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:52:59 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:52:59 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:52:59 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:52:59 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:52:59 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:52:59 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:52:59 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:00 fir-io2-s2 kernel: ll_ost03_026    S ffff90ff994c4100     0 84817      2 0x00000080
Apr 22 18:53:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:01 fir-io2-s2 kernel: ll_ost01_082    D ffff90ff994c5140     0 84818      2 0x00000080
Apr 22 18:53:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:01 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:53:01 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:53:01 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:53:01 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:53:01 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:53:01 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:53:01 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:53:01 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:01 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:01 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:01 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:02 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:02 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:02 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:02 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:02 fir-io2-s2 kernel: ll_ost01_083    D ffff90ff994c6180     0 84819      2 0x00000080
Apr 22 18:53:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:02 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:53:02 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:53:02 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:53:02 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:53:03 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:53:04 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:04 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:04 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:04 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:53:04 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:53:04 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:04 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:04 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:04 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:04 fir-io2-s2 kernel: ll_ost02_043    S ffff90ff98b18000     0 84820      2 0x00000080
Apr 22 18:53:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:05 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:53:05 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:53:05 fir-io2-s2 kernel: [<ffffffffc1209d70>] ldlm_completion_ast+0x430/0x890 [ptlrpc]
Apr 22 18:53:05 fir-io2-s2 kernel: [<ffffffffc1209720>] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc]
Apr 22 18:53:05 fir-io2-s2 kernel: [<ffffffffc11f0dcf>] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc]
Apr 22 18:53:05 fir-io2-s2 kernel: [<ffffffffc1207ef2>] ? ldlm_interval_alloc+0xa2/0x260 [ptlrpc]
Apr 22 18:53:05 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:05 fir-io2-s2 kernel: [<ffffffffc120ac8c>] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc]
Apr 22 18:53:05 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:53:05 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:53:05 fir-io2-s2 kernel: [<ffffffffc18016ad>] ofd_destroy_by_fid+0x1dd/0x4a0 [ofd]
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:07 fir-io2-s2 kernel: ll_ost01_084    S ffff90ff98b19040     0 84821      2 0x00000080
Apr 22 18:53:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:07 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:07 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:08 fir-io2-s2 kernel: ll_ost00_050    S ffff90ff98b1a080     0 84822      2 0x00000080
Apr 22 18:53:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:08 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:08 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:08 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:08 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:08 fir-io2-s2 kernel: ll_ost02_044    D ffff90ff98b1b0c0     0 84823      2 0x00000080
Apr 22 18:53:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:08 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:10 fir-io2-s2 kernel: ll_ost01_085    D ffff90ff98b1c100     0 84824      2 0x00000080
Apr 22 18:53:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:53:10 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:11 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:11 fir-io2-s2 kernel: ll_ost00_051    S ffff90ff98b1d140     0 84825      2 0x00000080
Apr 22 18:53:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:11 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:11 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:11 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:11 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:12 fir-io2-s2 kernel: ll_ost01_086    S ffff90ff98b1e180     0 84826      2 0x00000080
Apr 22 18:53:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:12 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 18:53:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:12 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:53:12 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:53:12 fir-io2-s2 kernel: [<ffffffffc1209d70>] ldlm_completion_ast+0x430/0x890 [ptlrpc]
Apr 22 18:53:12 fir-io2-s2 kernel: [<ffffffffc1209720>] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc]
Apr 22 18:53:12 fir-io2-s2 kernel: [<ffffffffc11f0dcf>] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc]
Apr 22 18:53:12 fir-io2-s2 kernel: [<ffffffffc1207ef2>] ? ldlm_interval_alloc+0xa2/0x260 [ptlrpc]
Apr 22 18:53:12 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:12 fir-io2-s2 kernel: [<ffffffffc120ac8c>] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc]
Apr 22 18:53:13 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:53:13 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:53:13 fir-io2-s2 kernel: [<ffffffffc18016ad>] ofd_destroy_by_fid+0x1dd/0x4a0 [ofd]
Apr 22 18:53:13 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:53:13 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:53:13 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:53:13 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:53:13 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:13 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:13 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:14 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:53:14 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:53:14 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:14 fir-io2-s2 kernel: ll_ost00_052    S ffff90ff799b0000     0 84827      2 0x00000080
Apr 22 18:53:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:14 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:14 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:14 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:15 fir-io2-s2 kernel: ll_ost01_087    S ffff90ff799b1040     0 84828      2 0x00000080
Apr 22 18:53:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:15 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:15 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:15 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:15 fir-io2-s2 kernel: Lustre: fir-OST000f: Export ffff90e1779acc00 already connecting from 10.8.26.4@o2ib6
Apr 22 18:53:15 fir-io2-s2 kernel: Lustre: Skipped 5 previous similar messages
Apr 22 18:53:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:16 fir-io2-s2 kernel: ll_ost03_027    S ffff90ff799b2080     0 84829      2 0x00000080
Apr 22 18:53:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:16 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:16 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:16 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:17 fir-io2-s2 kernel: ll_ost02_045    D ffff90ff799b30c0     0 84830      2 0x00000080
Apr 22 18:53:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:17 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:53:17 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:53:17 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:53:17 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:53:17 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:53:17 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:53:17 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:53:17 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:17 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:53:17 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:18 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:53:18 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:53:18 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:53:18 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:53:18 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:53:18 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:53:18 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:53:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:18 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:18 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:53:18 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:19 fir-io2-s2 kernel: ll_ost00_053    S ffff90ff799b4100     0 84831      2 0x00000080
Apr 22 18:53:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:20 fir-io2-s2 kernel: ll_ost01_088    S ffff90ff799b5140     0 84832      2 0x00000080
Apr 22 18:53:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:21 fir-io2-s2 kernel: ll_ost03_028    S ffff90ff799b6180     0 84833      2 0x00000080
Apr 22 18:53:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:21 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:21 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:21 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:21 fir-io2-s2 kernel: ll_ost00_054    S ffff90ff799a8000     0 84834      2 0x00000080
Apr 22 18:53:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:21 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:21 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:22 fir-io2-s2 kernel: ll_ost03_029    S ffff90ff799a9040     0 84835      2 0x00000080
Apr 22 18:53:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:23 fir-io2-s2 kernel: ll_ost01_089    D ffff90ff799aa080     0 84836      2 0x00000080
Apr 22 18:53:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:23 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:53:23 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:53:23 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:53:23 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:53:23 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:53:23 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:53:23 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:53:23 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:23 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:24 fir-io2-s2 kernel: ll_ost01_090    S ffff90ff799ab0c0     0 84837      2 0x00000080
Apr 22 18:53:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:25 fir-io2-s2 kernel: ll_ost01_091    D ffff90ff799ac100     0 84838      2 0x00000080
Apr 22 18:53:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:25 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:53:25 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:53:25 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:53:25 fir-io2-s2 kernel: [<ffffffffa90e3ce4>] ? find_busiest_group+0x144/0x990
Apr 22 18:53:25 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:53:25 fir-io2-s2 kernel: [<ffffffffa9381153>] ? number.isra.2+0x323/0x360
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffc15caac3>] ? ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffc15caac3>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffa92aed8a>] dqget+0x3fa/0x450
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffa92afb54>] dquot_get_dqblk+0x14/0x1f0
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffc16f8f55>] osd_acct_index_lookup+0x235/0x480 [osd_ldiskfs]
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffc165e80d>] lquotactl_slv+0x27d/0x9d0 [lquota]
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffc17f1a3c>] ofd_quotactl+0x13c/0x380 [ofd]
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:26 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:27 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:27 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:53:27 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:53:27 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:27 fir-io2-s2 kernel: ll_ost03_030    S ffff90ff799ad140     0 84839      2 0x00000080
Apr 22 18:53:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:28 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:28 fir-io2-s2 kernel: ll_ost02_046    D ffff90ff799ae180     0 84840      2 0x00000080
Apr 22 18:53:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:28 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:53:28 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:53:28 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:53:28 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:53:28 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:53:29 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:53:29 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:53:29 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:29 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:53:29 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:29 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:53:29 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:53:29 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:53:29 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:53:29 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:53:29 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:53:29 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:53:30 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:30 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:30 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:30 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:53:30 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:53:30 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:30 fir-io2-s2 kernel: ll_ost03_031    D ffff90fd0f138000     0 84841      2 0x00000080
Apr 22 18:53:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:30 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:53:31 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:53:31 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:53:31 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:53:31 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:53:31 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:53:31 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:53:31 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:31 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:31 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:31 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:53:31 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:53:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:32 fir-io2-s2 kernel: ll_ost01_092    S ffff90fd0f139040     0 84842      2 0x00000080
Apr 22 18:53:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:32 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:32 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:32 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:33 fir-io2-s2 kernel: ll_ost00_055    S ffff90fd0f13a080     0 84843      2 0x00000080
Apr 22 18:53:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:33 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:33 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:33 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:33 fir-io2-s2 kernel: ll_ost03_032    S ffff90fd0f13b0c0     0 84844      2 0x00000080
Apr 22 18:53:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:34 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:34 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:34 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:34 fir-io2-s2 kernel: ll_ost02_047    D ffff90fd0f13c100     0 84845      2 0x00000080
Apr 22 18:53:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:34 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:53:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:53:35 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:53:35 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:53:35 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:53:35 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:53:35 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:53:35 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:35 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:53:35 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:35 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:53:35 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:53:35 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:53:35 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:53:36 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:53:36 fir-io2-s2 kernel: Lustre: fir-OST000d: Export ffff90e188bb5400 already connecting from 10.9.106.36@o2ib4
Apr 22 18:53:36 fir-io2-s2 kernel: Lustre: Skipped 2 previous similar messages
Apr 22 18:53:36 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:53:36 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:53:36 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:36 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:36 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:36 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:36 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:36 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:37 fir-io2-s2 kernel: ll_ost01_093    D ffff90fd0f13d140     0 84846      2 0x00000080
Apr 22 18:53:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:37 fir-io2-s2 kernel: [<ffffffffc09e22d1>] ? jbd2_journal_dirty_metadata+0xd1/0x260 [jbd2]
Apr 22 18:53:37 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:53:37 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:53:37 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:53:37 fir-io2-s2 kernel: [<ffffffffc12977f5>] tgt_last_rcvd_update+0x295/0xc90 [ptlrpc]
Apr 22 18:53:37 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 18:53:37 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 18:53:37 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 18:53:37 fir-io2-s2 kernel: [<ffffffffc16d8f00>] ? osd_oi_insert+0x4d0/0x520 [osd_ldiskfs]
Apr 22 18:53:37 fir-io2-s2 kernel: Lustre: fir-OST0013: Client 48ad4899-3290-6782-9ecf-45ee04445b6e (at 10.8.1.27@o2ib6) reconnecting
Apr 22 18:53:37 fir-io2-s2 kernel: Lustre: Skipped 88 previous similar messages
Apr 22 18:53:38 fir-io2-s2 kernel: [<ffffffffc16c7edf>] ? osd_destroy+0x1af/0x760 [osd_ldiskfs]
Apr 22 18:53:38 fir-io2-s2 kernel: [<ffffffffc16c723d>] ? osd_ref_del+0x1ad/0x6a0 [osd_ldiskfs]
Apr 22 18:53:38 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:53:38 fir-io2-s2 kernel: [<ffffffffc1809455>] ofd_destroy+0x2c5/0x960 [ofd]
Apr 22 18:53:38 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:53:38 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:53:38 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:53:38 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:53:38 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:53:38 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:39 fir-io2-s2 kernel: ll_ost02_048    D ffff90fd0f13e180     0 84847      2 0x00000080
Apr 22 18:53:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:53:39 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:41 fir-io2-s2 kernel: ll_ost00_056    S ffff9102876b8000     0 84848      2 0x00000080
Apr 22 18:53:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:41 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:41 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:41 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:41 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:41 fir-io2-s2 kernel: ll_ost02_049    D ffff9102876b9040     0 84849      2 0x00000080
Apr 22 18:53:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:53:42 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:53:43 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:53:43 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:53:43 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:53:43 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:43 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:43 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:43 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:53:43 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:53:43 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:43 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:44 fir-io2-s2 kernel: ll_ost02_050    D ffff9102876ba080     0 84850      2 0x00000080
Apr 22 18:53:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:44 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:53:44 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:53:44 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:53:44 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:53:44 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:53:44 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:53:44 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:53:44 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:44 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:45 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:45 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:53:45 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:53:45 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:45 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:45 fir-io2-s2 kernel: ll_ost03_033    S ffff9102876bb0c0     0 84851      2 0x00000080
Apr 22 18:53:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:45 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:45 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:46 fir-io2-s2 kernel: ll_ost01_094    D ffff9102876bc100     0 84852      2 0x00000080
Apr 22 18:53:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffa93a0160>] ? __percpu_counter_sum+0x70/0x80
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:53:46 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:53:47 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:53:47 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:53:47 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:47 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:53:47 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:47 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 18:53:47 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 18:53:47 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 18:53:47 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 18:53:47 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:53:47 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 18:53:47 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:48 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:48 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:48 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:48 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:48 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:48 fir-io2-s2 kernel: ll_ost02_051    D ffff9102876bd140     0 84853      2 0x00000080
Apr 22 18:53:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:48 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:53:48 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:53:49 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:53:49 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:53:49 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:53:49 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:53:49 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:53:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:49 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:53:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:49 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:53:49 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:53:49 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:53:49 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:53:50 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:53:50 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:53:50 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:53:50 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:50 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:50 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:50 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:50 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:50 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:51 fir-io2-s2 kernel: ll_ost00_057    S ffff9102876be180     0 84854      2 0x00000080
Apr 22 18:53:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:51 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:51 fir-io2-s2 kernel: LustreError: 82371:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 165s: evicting client at 10.8.16.3@o2ib6  ns: filter-fir-OST0013_UUID lock: ffff90f143ec2880/0x4ec9aa14f8efba5f lrc: 3/0,0 mode: PW/PW res: [0x600000400:0x576c5f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->8392703) flags: 0x60000400010020 nid: 10.8.16.3@o2ib6 remote: 0x4d1ba193566af3eb expref: 18 pid: 83755 timeout: 30587 lvb_type: 0
Apr 22 18:53:51 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:51 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:51 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:52 fir-io2-s2 kernel: ll_ost01_095    S ffff91012d4b8000     0 84855      2 0x00000080
Apr 22 18:53:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:52 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:52 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:52 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:52 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:53 fir-io2-s2 kernel: ll_ost00_058    S ffff91012d4b9040     0 84856      2 0x00000080
Apr 22 18:53:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:53 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:53 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:53 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:53 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:53 fir-io2-s2 kernel: ll_ost01_096    S ffff91012d4ba080     0 84857      2 0x00000080
Apr 22 18:53:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:54 fir-io2-s2 kernel: ll_ost02_052    D ffff91012d4bb0c0     0 84858      2 0x00000080
Apr 22 18:53:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:53:54 fir-io2-s2 kernel: [<ffffffffa91c6e21>] ? truncate_inode_pages_range+0x301/0x700
Apr 22 18:53:55 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:53:55 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:53:55 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:53:55 fir-io2-s2 kernel: [<ffffffffc15b300f>] ? ldiskfs_evict_inode+0x16f/0x630 [ldiskfs]
Apr 22 18:53:55 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:53:55 fir-io2-s2 kernel: [<ffffffffc15b300f>] ldiskfs_evict_inode+0x16f/0x630 [ldiskfs]
Apr 22 18:53:55 fir-io2-s2 kernel: [<ffffffffa925eaa4>] evict+0xb4/0x180
Apr 22 18:53:55 fir-io2-s2 kernel: [<ffffffffa925f3ac>] iput+0xfc/0x190
Apr 22 18:53:55 fir-io2-s2 kernel: [<ffffffffc16bde3a>] osd_object_delete+0x1ea/0x340 [osd_ldiskfs]
Apr 22 18:53:55 fir-io2-s2 kernel: [<ffffffffc1054008>] lu_object_free.isra.30+0x68/0x170 [obdclass]
Apr 22 18:53:55 fir-io2-s2 kernel: [<ffffffffc0bfa378>] ? cfs_hash_bd_from_key+0x38/0xb0 [libcfs]
Apr 22 18:53:55 fir-io2-s2 kernel: [<ffffffffc10575d5>] lu_object_put+0xc5/0x3d0 [obdclass]
Apr 22 18:53:56 fir-io2-s2 kernel: [<ffffffffc18016ea>] ofd_destroy_by_fid+0x21a/0x4a0 [ofd]
Apr 22 18:53:56 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:53:56 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:53:56 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:53:56 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:53:56 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:53:56 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:53:56 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:53:56 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:56 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:53:56 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:57 fir-io2-s2 kernel: ll_ost01_097    S ffff91012d4bc100     0 84859      2 0x00000080
Apr 22 18:53:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:57 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:57 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:57 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:57 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:58 fir-io2-s2 kernel: ll_ost03_034    S ffff91012d4bd140     0 84860      2 0x00000080
Apr 22 18:53:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:58 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:53:58 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:53:58 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:53:58 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:53:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:53:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:53:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:53:58 fir-io2-s2 kernel: ll_ost01_098    D ffff91012d4be180     0 84861      2 0x00000080
Apr 22 18:53:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:53:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:53:58 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:53:58 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:53:59 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:53:59 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:53:59 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:53:59 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:53:59 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:53:59 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:59 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:53:59 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:53:59 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:53:59 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:53:59 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:53:59 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:01 fir-io2-s2 kernel: ll_ost00_059    D ffff91016c560000     0 84862      2 0x00000080
Apr 22 18:54:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:01 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:54:01 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:54:01 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:54:01 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:54:01 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:54:01 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:54:01 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:54:01 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:01 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:01 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:01 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:01 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:02 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:02 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:02 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:02 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:02 fir-io2-s2 kernel: ll_ost02_053    D ffff91016c561040     0 84863      2 0x00000080
Apr 22 18:54:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:02 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:54:02 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:54:02 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:54:02 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:54:03 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:54:03 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:54:03 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:54:03 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:03 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:54:03 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:03 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:54:03 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:54:03 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:54:03 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:03 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:03 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:54:04 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:54:04 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:04 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:04 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:04 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:04 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:04 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:04 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:04 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:04 fir-io2-s2 kernel: ll_ost00_060    S ffff91016c562080     0 84864      2 0x00000080
Apr 22 18:54:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:05 fir-io2-s2 kernel: ll_ost01_099    D ffff91016c5630c0     0 84865      2 0x00000080
Apr 22 18:54:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:54:05 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:54:06 fir-io2-s2 kernel: Lustre: fir-OST0011: Export ffff90ec535bf400 already connecting from 10.8.26.4@o2ib6
Apr 22 18:54:06 fir-io2-s2 kernel: Lustre: Skipped 2 previous similar messages
Apr 22 18:54:06 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:54:06 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:54:06 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:54:06 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:06 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:54:06 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:06 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:54:06 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:54:06 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:54:06 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:07 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:07 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:54:07 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:54:07 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:07 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:07 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:07 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:08 fir-io2-s2 kernel: ll_ost02_054    D ffff91016c564100     0 84866      2 0x00000080
Apr 22 18:54:08 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:08 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:54:08 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:54:08 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:54:08 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:54:08 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:54:08 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:54:08 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:54:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:08 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:54:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:09 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:54:09 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:54:09 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:54:09 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:09 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:09 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:54:09 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:54:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:09 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:09 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:09 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:09 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:10 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:10 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:10 fir-io2-s2 kernel: ll_ost03_035    D ffff91016c565140     0 84867      2 0x00000080
Apr 22 18:54:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:10 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:54:10 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:54:10 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:54:10 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:11 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:12 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:12 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:12 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:12 fir-io2-s2 kernel: ldlm_cn01_003   S ffff91016c566180     0 84868      2 0x00000080
Apr 22 18:54:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:13 fir-io2-s2 kernel: ldlm_cn02_003   S ffff90ff99408000     0 84869      2 0x00000080
Apr 22 18:54:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:14 fir-io2-s2 kernel: ll_ost02_055    D ffff90ff99409040     0 84870      2 0x00000080
Apr 22 18:54:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:14 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:54:14 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:54:14 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:54:14 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:54:14 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:54:14 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:54:14 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:54:14 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:15 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:15 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:15 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:15 fir-io2-s2 kernel: ll_ost00_061    S ffff90ff9940a080     0 84871      2 0x00000080
Apr 22 18:54:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:16 fir-io2-s2 kernel: ll_ost02_056    D ffff90ff9940b0c0     0 84872      2 0x00000080
Apr 22 18:54:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:54:16 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffa927882d>] ? __brelse+0x3d/0x50
Apr 22 18:54:17 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 18:54:18 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 18:54:18 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 18:54:18 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 18:54:18 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 18:54:18 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 18:54:18 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 18:54:18 fir-io2-s2 kernel: [<ffffffffc16d8f00>] ? osd_oi_insert+0x4d0/0x520 [osd_ldiskfs]
Apr 22 18:54:18 fir-io2-s2 kernel: [<ffffffffc16c7edf>] ? osd_destroy+0x1af/0x760 [osd_ldiskfs]
Apr 22 18:54:18 fir-io2-s2 kernel: [<ffffffffc16c723d>] ? osd_ref_del+0x1ad/0x6a0 [osd_ldiskfs]
Apr 22 18:54:18 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:54:18 fir-io2-s2 kernel: [<ffffffffc1809455>] ofd_destroy+0x2c5/0x960 [ofd]
Apr 22 18:54:18 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:54:19 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:19 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:19 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:54:19 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:54:19 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:19 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:19 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:19 fir-io2-s2 kernel: [<ffffffffa976a175>] ? _raw_spin_unlock_irqrestore+0x15/0x20
Apr 22 18:54:19 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:20 fir-io2-s2 kernel: ll_ost03_036    D ffff90ff9940c100     0 84873      2 0x00000080
Apr 22 18:54:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:20 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 18:54:21 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 18:54:21 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 18:54:21 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 18:54:21 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:54:21 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 18:54:21 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:21 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:21 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:21 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:21 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:21 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:22 fir-io2-s2 kernel: ll_ost00_062    S ffff90ff9940d140     0 84874      2 0x00000080
Apr 22 18:54:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:22 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:22 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:22 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:22 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:23 fir-io2-s2 kernel: ll_ost00_063    S ffff90ff9940e180     0 84875      2 0x00000080
Apr 22 18:54:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:23 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:23 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:23 fir-io2-s2 kernel: ll_ost00_064    D ffff90fd19948000     0 84876      2 0x00000080
Apr 22 18:54:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:23 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa93824d9>] ? pointer.isra.19+0x1c9/0x4d0
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:54:24 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 18:54:25 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:54:25 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:54:25 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 18:54:25 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 18:54:25 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:25 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 18:54:25 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 18:54:25 fir-io2-s2 kernel: [<ffffffffc1298909>] tgt_client_data_update+0x3a9/0x5e0 [ptlrpc]
Apr 22 18:54:25 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 18:54:25 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 18:54:25 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 18:54:25 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:54:26 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 18:54:26 fir-io2-s2 kernel: Lustre: fir-OST0011: Export ffff90e29cb49400 already connecting from 10.9.106.36@o2ib4
Apr 22 18:54:26 fir-io2-s2 kernel: Lustre: Skipped 2 previous similar messages
Apr 22 18:54:26 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:26 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:26 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:26 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:26 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:26 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:27 fir-io2-s2 kernel: ll_ost02_057    S ffff90fd19949040     0 84877      2 0x00000080
Apr 22 18:54:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:27 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 18:54:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:27 fir-io2-s2 kernel: [<ffffffffa9765668>] schedule_timeout+0x168/0x2d0
Apr 22 18:54:27 fir-io2-s2 kernel: [<ffffffffa90a98e0>] ? __internal_add_timer+0x130/0x130
Apr 22 18:54:27 fir-io2-s2 kernel: [<ffffffffc1209d70>] ldlm_completion_ast+0x430/0x890 [ptlrpc]
Apr 22 18:54:27 fir-io2-s2 kernel: [<ffffffffc1209720>] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc]
Apr 22 18:54:27 fir-io2-s2 kernel: [<ffffffffc11f0dcf>] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc]
Apr 22 18:54:27 fir-io2-s2 kernel: [<ffffffffc1207ef2>] ? ldlm_interval_alloc+0xa2/0x260 [ptlrpc]
Apr 22 18:54:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:27 fir-io2-s2 kernel: [<ffffffffc120ac8c>] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc]
Apr 22 18:54:28 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:28 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:28 fir-io2-s2 kernel: [<ffffffffc18016ad>] ofd_destroy_by_fid+0x1dd/0x4a0 [ofd]
Apr 22 18:54:28 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:28 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:28 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:54:28 fir-io2-s2 kernel: LustreError: 84922:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555984168, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-fir-OST000d_UUID lock: ffff90f64b1d69c0/0x4ec9aa14f8ee5d2a lrc: 3/0,1 mode: --/PW res: [0x300000400:0x576a7d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 84922 timeout: 0 lvb_type: 0
Apr 22 18:54:28 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984468.84922
Apr 22 18:54:29 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:54:29 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:29 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:29 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:29 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:29 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:29 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:30 fir-io2-s2 kernel: ll_ost03_037    S ffff90fd1994a080     0 84878      2 0x00000080
Apr 22 18:54:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:30 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:30 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:30 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:30 fir-io2-s2 kernel: ll_ost02_058    D ffff90fd1994b0c0     0 84879      2 0x00000080
Apr 22 18:54:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:31 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:54:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:54:31 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:54:31 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:54:31 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:54:31 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:54:31 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:54:31 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:31 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:54:31 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:31 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:54:31 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:33 fir-io2-s2 kernel: ll_ost00_065    S ffff90fd1994c100     0 84880      2 0x00000080
Apr 22 18:54:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:33 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:33 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:33 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:34 fir-io2-s2 kernel: ll_ost02_059    D ffff90fd1994d140     0 84881      2 0x00000080
Apr 22 18:54:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:34 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:54:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:54:34 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:54:34 fir-io2-s2 kernel: [<ffffffffa9767eea>] ? _cond_resched+0x3a/0x50
Apr 22 18:54:34 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:54:34 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:54:34 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:54:34 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:54:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:34 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:54:35 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:35 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:54:35 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:54:35 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:54:35 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:35 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:35 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:54:35 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:54:35 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:35 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:35 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:36 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:36 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:36 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:36 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:36 fir-io2-s2 kernel: ll_ost03_038    S ffff90fd1994e180     0 84882      2 0x00000080
Apr 22 18:54:36 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:36 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:36 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:36 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:37 fir-io2-s2 kernel: ll_ost00_066    S ffff90fd0f1d8000     0 84883      2 0x00000080
Apr 22 18:54:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:38 fir-io2-s2 kernel: ll_ost00_067    D ffff90fd0f1d9040     0 84884      2 0x00000080
Apr 22 18:54:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:38 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:54:38 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:54:38 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:54:38 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:54:38 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:54:38 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:38 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:38 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:38 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:38 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:38 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:39 fir-io2-s2 kernel: ll_ost02_060    D ffff90fd0f1da080     0 84885      2 0x00000080
Apr 22 18:54:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:54:39 fir-io2-s2 kernel: LNet: Service thread pid 82397 was inactive for 200.74s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Apr 22 18:54:39 fir-io2-s2 kernel: Pid: 82397, comm: ll_ost_create01 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 18:54:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:39 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 18:54:39 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984479.82397
Apr 22 18:54:40 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:54:40 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:54:41 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:41 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:54:41 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:41 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:54:41 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:54:41 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:54:41 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:41 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:41 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:54:41 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:54:41 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:41 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:42 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:42 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:42 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:42 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:42 fir-io2-s2 kernel: ll_ost03_039    S ffff90fd0f1db0c0     0 84886      2 0x00000080
Apr 22 18:54:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:42 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:42 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:43 fir-io2-s2 kernel: ll_ost00_068    S ffff90fd0f1dc100     0 84887      2 0x00000080
Apr 22 18:54:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:43 fir-io2-s2 kernel: LNet: Service thread pid 82399 was inactive for 200.44s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Apr 22 18:54:43 fir-io2-s2 kernel: Pid: 82399, comm: ll_ost_create02 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 18:54:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 18:54:43 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984483.82399
Apr 22 18:54:43 fir-io2-s2 kernel: Pid: 82400, comm: ll_ost_create02 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 18:54:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:43 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 18:54:45 fir-io2-s2 kernel: [<ffffffffa90d0884>] ? finish_task_switch+0x54/0x1c0
Apr 22 18:54:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:46 fir-io2-s2 kernel: ll_ost02_061    D ffff90fd0f1dd140     0 84888      2 0x00000080
Apr 22 18:54:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:54:46 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:54:47 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:54:47 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:47 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:47 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:54:47 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:54:47 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:47 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:47 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:47 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:47 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:47 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:47 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:48 fir-io2-s2 kernel: ll_ost02_062    D ffff90fd0f1de180     0 84889      2 0x00000080
Apr 22 18:54:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:48 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:54:48 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:54:48 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:54:48 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:54:48 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:54:48 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:49 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:50 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:50 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:50 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:50 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:50 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:50 fir-io2-s2 kernel: ll_ost00_069    S ffff90ff36c48000     0 84890      2 0x00000080
Apr 22 18:54:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:50 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:50 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:51 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:51 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:51 fir-io2-s2 kernel: ll_ost00_070    S ffff90ff36c49040     0 84891      2 0x00000080
Apr 22 18:54:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:51 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:51 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:51 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:51 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:52 fir-io2-s2 kernel: ll_ost03_040    S ffff90ff36c4a080     0 84892      2 0x00000080
Apr 22 18:54:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:53 fir-io2-s2 kernel: ll_ost02_063    D ffff90ff36c4b0c0     0 84893      2 0x00000080
Apr 22 18:54:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:53 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:54:53 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:54:53 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:54:53 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:54:53 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:54:53 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:54:53 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:54:53 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:53 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:54:53 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:54 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:54:54 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:54:54 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:54:54 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:54:54 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:54:54 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:54:54 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:54:54 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:54:54 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:54 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:54:54 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:54:54 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:55 fir-io2-s2 kernel: ll_ost00_071    S ffff90ff36c4c100     0 84894      2 0x00000080
Apr 22 18:54:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:56 fir-io2-s2 kernel: Lustre: fir-OST000d: Export ffff90ef4dda5000 already connecting from 10.8.26.4@o2ib6
Apr 22 18:54:56 fir-io2-s2 kernel: Lustre: Skipped 3 previous similar messages
Apr 22 18:54:56 fir-io2-s2 kernel: ll_ost00_072    S ffff90ff36c4d140     0 84895      2 0x00000080
Apr 22 18:54:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:56 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:54:56 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:54:56 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:54:56 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:54:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:57 fir-io2-s2 kernel: LustreError: 84820:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555984196, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-fir-OST0011_UUID lock: ffff910201ec3a80/0x4ec9aa14f8f268c8 lrc: 3/0,1 mode: --/PW res: [0x680000400:0x576abf:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 84820 timeout: 0 lvb_type: 0
Apr 22 18:54:58 fir-io2-s2 kernel: LNet: Service thread pid 45403 was inactive for 200.09s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Apr 22 18:54:58 fir-io2-s2 kernel: LNet: Skipped 1 previous similar message
Apr 22 18:54:58 fir-io2-s2 kernel: Pid: 45403, comm: ll_ost_create02 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 18:54:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:58 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:54:58 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:54:58 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:54:58 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:54:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:54:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:54:58 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 18:54:58 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984497.45403
Apr 22 18:54:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:54:58 fir-io2-s2 kernel: ll_ost02_064    D ffff90ff36c4e180     0 84896      2 0x00000080
Apr 22 18:54:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:54:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:54:58 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:54:58 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:54:59 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:54:59 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:54:59 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:54:59 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:54:59 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:54:59 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:59 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:54:59 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:54:59 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:54:59 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:54:59 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:54:59 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:01 fir-io2-s2 kernel: ll_ost03_041    S ffff90feb1980000     0 84897      2 0x00000080
Apr 22 18:55:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:01 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:01 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:01 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:01 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:01 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:01 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:01 fir-io2-s2 kernel: ll_ost02_065    D ffff90feb1981040     0 84898      2 0x00000080
Apr 22 18:55:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:55:02 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:55:03 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:55:03 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:55:03 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:55:03 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:03 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:03 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:03 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:03 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:03 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:03 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:03 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:04 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:04 fir-io2-s2 kernel: ldlm_cn01_004   S ffff90feb1982080     0 84899      2 0x00000080
Apr 22 18:55:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:04 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:04 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:04 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:04 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:04 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:04 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:05 fir-io2-s2 kernel: ll_ost00_073    S ffff90feb19830c0     0 84900      2 0x00000080
Apr 22 18:55:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:05 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:05 fir-io2-s2 kernel: [<ffffffffa976a175>] ? _raw_spin_unlock_irqrestore+0x15/0x20
Apr 22 18:55:05 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:05 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:05 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:05 fir-io2-s2 kernel: ldlm_cn00_004   S ffff90feb1984100     0 84901      2 0x00000080
Apr 22 18:55:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:06 fir-io2-s2 kernel: ll_ost03_042    S ffff90feb1985140     0 84902      2 0x00000080
Apr 22 18:55:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:06 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:07 fir-io2-s2 kernel: ll_ost02_066    D ffff90feb1986180     0 84903      2 0x00000080
Apr 22 18:55:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:07 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:55:07 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:55:07 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:55:07 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:55:07 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:55:07 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:55:08 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:55:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:08 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:55:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:08 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:55:08 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:55:08 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:55:08 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:55:08 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:55:08 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:55:08 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:09 fir-io2-s2 kernel: ll_ost00_074    S ffff91014a5e8000     0 84904      2 0x00000080
Apr 22 18:55:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:09 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:10 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:10 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:10 fir-io2-s2 kernel: ll_ost03_043    S ffff91014a5e9040     0 84905      2 0x00000080
Apr 22 18:55:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:10 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:10 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:10 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:10 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:11 fir-io2-s2 kernel: ll_ost02_067    D ffff91014a5ea080     0 84906      2 0x00000080
Apr 22 18:55:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:55:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:55:12 fir-io2-s2 kernel: LNet: Service thread pid 84922 was inactive for 436.21s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Apr 22 18:55:12 fir-io2-s2 kernel: Pid: 84922, comm: ll_ost02_072 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 18:55:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc1209e25>] ldlm_completion_ast+0x4e5/0x890 [ptlrpc]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc120ac8c>] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc18016ad>] ofd_destroy_by_fid+0x1dd/0x4a0 [ofd]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:12 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 18:55:12 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984512.84922
Apr 22 18:55:13 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:55:13 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:14 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:14 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:14 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:14 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:14 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:14 fir-io2-s2 kernel: ll_ost_io02_004 D ffff91014a5eb0c0     0 84907      2 0x00000080
Apr 22 18:55:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:15 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:55:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:55:15 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:55:15 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:55:15 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:55:15 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:55:15 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:55:15 fir-io2-s2 kernel: LustreError: 84909:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555984215, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-fir-OST0015_UUID lock: ffff90f368adb180/0x4ec9aa14f8f811e4 lrc: 3/0,1 mode: --/PW res: [0x740000400:0x5772b2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 84909 timeout: 0 lvb_type: 0
Apr 22 18:55:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:16 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:55:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:16 fir-io2-s2 kernel: Lustre: fir-OST000f: Export ffff91126541e800 already connecting from 10.9.106.36@o2ib4
Apr 22 18:55:16 fir-io2-s2 kernel: Lustre: Skipped 2 previous similar messages
Apr 22 18:55:16 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:55:16 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:55:16 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:55:16 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:55:16 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:55:16 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:18 fir-io2-s2 kernel: ll_ost00_075    D ffff91014a5ec100     0 84908      2 0x00000080
Apr 22 18:55:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:18 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:55:18 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:55:18 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:55:18 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:55:18 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:55:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:18 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:18 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:18 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:18 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:18 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:19 fir-io2-s2 kernel: ll_ost02_068    S ffff91014a5ed140     0 84909      2 0x00000080
Apr 22 18:55:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:19 fir-io2-s2 kernel: [<ffffffffc0beec05>] ? ktime_get_seconds+0x25/0x40 [libcfs]
Apr 22 18:55:19 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:55:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:19 fir-io2-s2 kernel: [<ffffffffc1209e25>] ldlm_completion_ast+0x4e5/0x890 [ptlrpc]
Apr 22 18:55:19 fir-io2-s2 kernel: [<ffffffffc1209720>] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc]
Apr 22 18:55:19 fir-io2-s2 kernel: [<ffffffffc11f0dcf>] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc]
Apr 22 18:55:19 fir-io2-s2 kernel: [<ffffffffc1207ef2>] ? ldlm_interval_alloc+0xa2/0x260 [ptlrpc]
Apr 22 18:55:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:20 fir-io2-s2 kernel: [<ffffffffc120ac8c>] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc]
Apr 22 18:55:20 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:55:20 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:55:20 fir-io2-s2 kernel: [<ffffffffc18016ad>] ofd_destroy_by_fid+0x1dd/0x4a0 [ofd]
Apr 22 18:55:20 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:55:20 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:55:20 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:55:20 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:20 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:20 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:21 fir-io2-s2 kernel: ll_ost00_076    S ffff91014a5ee180     0 84910      2 0x00000080
Apr 22 18:55:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:22 fir-io2-s2 kernel: ll_ost03_044    D ffff9100c8168000     0 84911      2 0x00000080
Apr 22 18:55:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:22 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:55:22 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:55:22 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:55:22 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:55:22 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:55:22 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:55:22 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:55:22 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:23 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:23 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:23 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:23 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:23 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:23 fir-io2-s2 kernel: ll_ost00_077    D ffff9100c8169040     0 84912      2 0x00000080
Apr 22 18:55:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:23 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:25 fir-io2-s2 kernel: ll_ost02_069    D ffff9100c816a080     0 84913      2 0x00000080
Apr 22 18:55:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:25 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:55:25 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:55:25 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:55:25 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:55:25 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:55:25 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:55:25 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:55:25 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:26 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:27 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:27 fir-io2-s2 kernel: ll_ost03_045    S ffff9100c816b0c0     0 84914      2 0x00000080
Apr 22 18:55:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:27 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:27 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:27 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:28 fir-io2-s2 kernel: ll_ost02_070    D ffff9100c816c100     0 84915      2 0x00000080
Apr 22 18:55:28 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:28 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:29 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:29 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:29 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:29 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:29 fir-io2-s2 kernel: ll_ost_io02_005 D ffff9100c816d140     0 84916      2 0x00000080
Apr 22 18:55:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:55:30 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:55:31 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:55:31 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:55:31 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:55:31 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:55:31 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:55:31 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:31 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:31 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:31 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:31 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:31 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:32 fir-io2-s2 kernel: ll_ost03_046    S ffff9100c816e180     0 84917      2 0x00000080
Apr 22 18:55:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:33 fir-io2-s2 kernel: ll_ost02_071    D ffff90ff576a8000     0 84918      2 0x00000080
Apr 22 18:55:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:33 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:55:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:55:33 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:55:33 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:55:33 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:55:33 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:55:33 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:55:33 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:33 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:55:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:34 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:55:34 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:55:34 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:55:34 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:55:34 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:55:34 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:55:34 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:34 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:34 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:34 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:34 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:35 fir-io2-s2 kernel: ll_ost_io02_006 D ffff90ff576a9040     0 84919      2 0x00000080
Apr 22 18:55:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:55:35 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:55:36 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:55:36 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:55:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:36 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:55:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:36 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:55:36 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:55:36 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:55:36 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:55:36 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:55:36 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:55:36 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:55:37 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:55:37 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:37 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:37 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:37 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:37 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:37 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:37 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:38 fir-io2-s2 kernel: ll_ost03_047    S ffff90ff576aa080     0 84920      2 0x00000080
Apr 22 18:55:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:38 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:38 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:38 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:38 fir-io2-s2 kernel: ll_ost00_078    S ffff90ff576ab0c0     0 84921      2 0x00000080
Apr 22 18:55:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:39 fir-io2-s2 kernel: ll_ost02_072    S ffff90ff576ac100     0 84922      2 0x00000080
Apr 22 18:55:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffa90ac4ab>] ? recalc_sigpending+0x1b/0x70
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffc1209e25>] ldlm_completion_ast+0x4e5/0x890 [ptlrpc]
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffc1209720>] ? ldlm_completion_ast_async+0x300/0x300 [ptlrpc]
Apr 22 18:55:39 fir-io2-s2 kernel: [<ffffffffc11f0dcf>] ? ldlm_lock_enqueue+0x40f/0xa60 [ptlrpc]
Apr 22 18:55:40 fir-io2-s2 kernel: [<ffffffffc1207ef2>] ? ldlm_interval_alloc+0xa2/0x260 [ptlrpc]
Apr 22 18:55:40 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:40 fir-io2-s2 kernel: [<ffffffffc120ac8c>] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc]
Apr 22 18:55:40 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:55:40 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:55:40 fir-io2-s2 kernel: [<ffffffffc18016ad>] ofd_destroy_by_fid+0x1dd/0x4a0 [ofd]
Apr 22 18:55:40 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:55:40 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:55:40 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:55:40 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:40 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:41 fir-io2-s2 kernel: ll_ost03_048    S ffff90ff576ad140     0 84923      2 0x00000080
Apr 22 18:55:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:41 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:42 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:42 fir-io2-s2 kernel: ll_ost00_079    D ffff90ff576ae180     0 84924      2 0x00000080
Apr 22 18:55:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:42 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:55:42 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:55:42 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:55:42 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:55:42 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:55:43 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:55:43 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:55:43 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:43 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:43 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:43 fir-io2-s2 kernel: LNet: Service thread pid 82388 was inactive for 286.90s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 18:55:43 fir-io2-s2 kernel: LNet: Skipped 6 previous similar messages
Apr 22 18:55:43 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984543.82388
Apr 22 18:55:43 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:43 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:43 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:43 fir-io2-s2 kernel: LustreError: 84826:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555984243, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-fir-OST000d_UUID lock: ffff90f06c7e72c0/0x4ec9aa14f8fe3f43 lrc: 3/0,1 mode: --/PW res: [0x300000402:0x929319:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 84826 timeout: 0 lvb_type: 0
Apr 22 18:55:44 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:44 fir-io2-s2 kernel: ll_ost02_073    D ffff90ff57618000     0 84925      2 0x00000080
Apr 22 18:55:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:44 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 18:55:45 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:55:46 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:55:46 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 18:55:46 fir-io2-s2 kernel: [<ffffffffa927882d>] ? __brelse+0x3d/0x50
Apr 22 18:55:46 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 18:55:46 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 18:55:46 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 18:55:46 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 18:55:46 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 18:55:46 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 18:55:46 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 18:55:46 fir-io2-s2 kernel: [<ffffffffc16d8f00>] ? osd_oi_insert+0x4d0/0x520 [osd_ldiskfs]
Apr 22 18:55:46 fir-io2-s2 kernel: [<ffffffffc16c7edf>] ? osd_destroy+0x1af/0x760 [osd_ldiskfs]
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffc16c723d>] ? osd_ref_del+0x1ad/0x6a0 [osd_ldiskfs]
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffc1809455>] ofd_destroy+0x2c5/0x960 [ofd]
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:47 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:48 fir-io2-s2 kernel: ll_ost00_080    S ffff90ff57619040     0 84926      2 0x00000080
Apr 22 18:55:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:49 fir-io2-s2 kernel: ll_ost02_074    S ffff90ff5761a080     0 84927      2 0x00000080
Apr 22 18:55:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:49 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:49 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:49 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:49 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:50 fir-io2-s2 kernel: ll_ost00_081    S ffff90ff5761b0c0     0 84928      2 0x00000080
Apr 22 18:55:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:50 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:50 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:50 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:50 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:50 fir-io2-s2 kernel: ll_ost03_049    S ffff90ff5761c100     0 84929      2 0x00000080
Apr 22 18:55:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:50 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:55:50 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:55:51 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:55:51 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:51 fir-io2-s2 kernel: ll_ost00_082    D ffff90ff5761d140     0 84930      2 0x00000080
Apr 22 18:55:51 fir-io2-s2 kernel: LNet: Service thread pid 83775 was inactive for 258.86s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 18:55:51 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984551.83775
Apr 22 18:55:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:51 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:55:51 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:55:52 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:55:52 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:55:52 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:55:52 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:55:52 fir-io2-s2 kernel: Lustre: fir-OST000f: Export ffff9111fe6d1c00 already connecting from 10.9.104.58@o2ib4
Apr 22 18:55:52 fir-io2-s2 kernel: Lustre: Skipped 4 previous similar messages
Apr 22 18:55:52 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:55:52 fir-io2-s2 kernel: LNet: Service thread pid 82425 was inactive for 286.12s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 18:55:52 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984552.82425
Apr 22 18:55:52 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:52 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:55:53 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:53 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 18:55:53 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 18:55:53 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 18:55:53 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 18:55:53 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:55:53 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 18:55:53 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:53 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984553.84846
Apr 22 18:55:53 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:53 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:53 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:53 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:54 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:54 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:54 fir-io2-s2 kernel: ll_ost02_075    D ffff90ff5761e180     0 84931      2 0x00000080
Apr 22 18:55:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:54 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:55:54 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:55:54 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:55:54 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:55:55 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:55:55 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:55:55 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:55:55 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:55 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:55:55 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:55 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:55:55 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:55:55 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:55:55 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:55:55 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:55:55 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:55:56 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:56 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:56 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:56 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:56 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:56 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:56 fir-io2-s2 kernel: LustreError: 82388:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555984256, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-fir-OST0013_UUID lock: ffff90f7f2e23cc0/0x4ec9aa14f904a6d9 lrc: 3/0,1 mode: --/PW res: [0x600000400:0x576beb:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 82388 timeout: 0 lvb_type: 0
Apr 22 18:55:57 fir-io2-s2 kernel: LNet: Service thread pid 83782 was inactive for 274.61s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 18:55:57 fir-io2-s2 kernel: LNet: Skipped 2 previous similar messages
Apr 22 18:55:57 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984556.83782
Apr 22 18:55:57 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:57 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:55:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:57 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984557.84820
Apr 22 18:55:57 fir-io2-s2 kernel: ll_ost02_076    D ffff90ff36c68000     0 84932      2 0x00000080
Apr 22 18:55:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:55:57 fir-io2-s2 kernel: [<ffffffffc0c5aff2>] ? fld_local_lookup+0x62/0x270 [fld]
Apr 22 18:55:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:55:58 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:55:58 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:55:58 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:55:58 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:55:58 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:55:58 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:55:58 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:55:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:58 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:55:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:55:58 fir-io2-s2 kernel: [<ffffffffc1805f17>] ofd_precreate_objects+0xa57/0x1ab0 [ofd]
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffc17f8794>] ofd_create_hdl+0x474/0x20e0 [ofd]
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffc123f4b7>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:55:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:00 fir-io2-s2 kernel: ll_ost00_083    S ffff90ff36c69040     0 84933      2 0x00000080
Apr 22 18:56:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:00 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:56:00 fir-io2-s2 kernel: [<ffffffffa976a175>] ? _raw_spin_unlock_irqrestore+0x15/0x20
Apr 22 18:56:00 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:56:00 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:56:00 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:00 fir-io2-s2 kernel: LNet: Service thread pid 83789 was inactive for 258.18s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 18:56:00 fir-io2-s2 kernel: LNet: Skipped 1 previous similar message
Apr 22 18:56:00 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984560.83789
Apr 22 18:56:01 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:01 fir-io2-s2 kernel: ll_ost03_050    D ffff90ff36c6a080     0 84934      2 0x00000080
Apr 22 18:56:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:01 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:56:01 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:01 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:56:01 fir-io2-s2 kernel: [<ffffffffa93a0160>] ? __percpu_counter_sum+0x70/0x80
Apr 22 18:56:01 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:56:01 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:56:01 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:56:01 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:02 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:02 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:56:02 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:02 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 18:56:02 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 18:56:02 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 18:56:02 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 18:56:02 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 18:56:02 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:02 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:02 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:02 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:03 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:03 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:03 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:03 fir-io2-s2 kernel: ll_ost_io02_007 D ffff90ff36c6b0c0     0 84935      2 0x00000080
Apr 22 18:56:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:03 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:56:03 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:03 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:56:03 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:56:04 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:56:04 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:56:04 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:04 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:04 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:56:04 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:04 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:56:04 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:56:04 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:56:04 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:56:04 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:56:04 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:06 fir-io2-s2 kernel: ll_ost03_051    S ffff90ff36c6c100     0 84936      2 0x00000080
Apr 22 18:56:06 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:06 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:56:06 fir-io2-s2 kernel: [<ffffffffa976a470>] ? _raw_spin_lock+0x10/0x30
Apr 22 18:56:06 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:56:06 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:56:06 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:07 fir-io2-s2 kernel: ll_ost00_084    S ffff90ff36c6d140     0 84937      2 0x00000080
Apr 22 18:56:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:07 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:56:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:56:07 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:56:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:07 fir-io2-s2 kernel: ll_ost02_077    D ffff90ff36c6e180     0 84938      2 0x00000080
Apr 22 18:56:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:07 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:56:07 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984567.84720
Apr 22 18:56:08 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:08 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:56:08 fir-io2-s2 kernel: [<ffffffffc16bd200>] ? osd_shutdown+0x2e0/0x2e0 [osd_ldiskfs]
Apr 22 18:56:08 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:56:08 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:56:08 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:08 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:56:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:08 fir-io2-s2 kernel: [<ffffffffc1805f17>] ofd_precreate_objects+0xa57/0x1ab0 [ofd]
Apr 22 18:56:08 fir-io2-s2 kernel: [<ffffffffc17f8794>] ofd_create_hdl+0x474/0x20e0 [ofd]
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffc123f4b7>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:09 fir-io2-s2 kernel: LNet: Service thread pid 84731 was inactive for 274.32s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 18:56:09 fir-io2-s2 kernel: LNet: Skipped 21 previous similar messages
Apr 22 18:56:09 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984569.84731
Apr 22 18:56:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:10 fir-io2-s2 kernel: ll_ost00_085    D ffff90ff33bd0000     0 84939      2 0x00000080
Apr 22 18:56:10 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:10 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:56:10 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:10 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:56:10 fir-io2-s2 kernel: [<ffffffffa921a849>] ? ___slab_alloc+0x209/0x4f0
Apr 22 18:56:10 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:56:10 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:56:10 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984570.83745
Apr 22 18:56:11 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:56:11 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:11 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:56:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:11 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 18:56:11 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 18:56:11 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 18:56:11 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 18:56:11 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:11 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 18:56:11 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:12 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:12 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:12 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:12 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:12 fir-io2-s2 kernel: ll_ost03_052    S ffff90ff33bd1040     0 84940      2 0x00000080
Apr 22 18:56:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:12 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:56:13 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984573.84913
Apr 22 18:56:13 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:56:13 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:56:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:13 fir-io2-s2 kernel: ll_ost_io01_004 D ffff90ff33bd2080     0 84941      2 0x00000080
Apr 22 18:56:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:13 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 18:56:13 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:14 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:15 fir-io2-s2 kernel: ll_ost02_078    D ffff90ff33bd30c0     0 84942      2 0x00000080
Apr 22 18:56:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:15 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:56:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:15 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:56:15 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:56:16 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:56:16 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:56:16 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:16 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:56:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:16 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:56:16 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:56:16 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:56:16 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:56:16 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:56:16 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:56:17 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:17 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:17 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984577.84750
Apr 22 18:56:17 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:18 fir-io2-s2 kernel: ll_ost03_053    S ffff90ff33bd4100     0 84943      2 0x00000080
Apr 22 18:56:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:18 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984578.84838
Apr 22 18:56:18 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:56:18 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:56:18 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:56:18 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:19 fir-io2-s2 kernel: ll_ost00_086    D ffff90ff33bd5140     0 84944      2 0x00000080
Apr 22 18:56:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:20 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984580.84826
Apr 22 18:56:20 fir-io2-s2 kernel: ll_ost03_054    S ffff90ff33bd6180     0 84945      2 0x00000080
Apr 22 18:56:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:56:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:56:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:56:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:21 fir-io2-s2 kernel: ll_ost00_087    S ffff90ff33bf8000     0 84946      2 0x00000080
Apr 22 18:56:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:21 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:56:21 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:56:21 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:56:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:21 fir-io2-s2 kernel: ll_ost03_055    S ffff90ff33bf9040     0 84947      2 0x00000080
Apr 22 18:56:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:22 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:56:22 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:56:22 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:56:22 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:22 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984582.84779
Apr 22 18:56:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:22 fir-io2-s2 kernel: ll_ost02_079    D ffff90ff33bfa080     0 84948      2 0x00000080
Apr 22 18:56:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:22 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:56:22 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:24 fir-io2-s2 kernel: ll_ost_io01_005 D ffff90ff33bfb0c0     0 84949      2 0x00000080
Apr 22 18:56:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:24 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:56:24 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:24 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984584.82383
Apr 22 18:56:24 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:56:24 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:56:24 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:56:24 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:56:24 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:24 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:24 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:56:25 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:25 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:56:25 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:56:25 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:56:25 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:56:25 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:56:25 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:56:25 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:56:25 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:25 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:56:25 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:25 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:26 fir-io2-s2 kernel: ll_ost_io00_004 D ffff90ff33bfc100     0 84950      2 0x00000080
Apr 22 18:56:26 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 18:56:26 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 18:56:27 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:27 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:56:27 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 18:56:27 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:56:27 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:56:27 fir-io2-s2 kernel: LNet: Service thread pid 85080 was inactive for 262.44s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 18:56:27 fir-io2-s2 kernel: LNet: Skipped 21 previous similar messages
Apr 22 18:56:27 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984587.85080
Apr 22 18:56:27 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:56:27 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:56:27 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:56:27 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:28 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:56:28 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:28 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:28 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:28 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:28 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:28 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:28 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:29 fir-io2-s2 kernel: ll_ost_io03_004 D ffff90ff33bfd140     0 84951      2 0x00000080
Apr 22 18:56:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:29 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:56:29 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:29 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:56:29 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:56:29 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984589.83743
Apr 22 18:56:29 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:56:29 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:56:29 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:29 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:29 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:56:29 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:30 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:56:30 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:56:30 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:56:30 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:56:30 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:56:30 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:56:30 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:56:30 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:30 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:56:30 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:30 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:30 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:31 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:31 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:31 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:31 fir-io2-s2 kernel: ll_ost_io02_008 D ffff90ff33bfe180     0 84952      2 0x00000080
Apr 22 18:56:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:31 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:56:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:32 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:56:32 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:56:32 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:56:32 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:56:32 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:32 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:32 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:56:32 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:32 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:56:32 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:56:32 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:56:32 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:56:33 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:56:33 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:56:33 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:56:33 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:33 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:56:33 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:33 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:33 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:33 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:33 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:33 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:33 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:34 fir-io2-s2 kernel: ll_ost02_080    D ffff90fd118f0000     0 84953      2 0x00000080
Apr 22 18:56:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:34 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:35 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:35 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:35 fir-io2-s2 kernel: [<ffffffffa90d0884>] ? finish_task_switch+0x54/0x1c0
Apr 22 18:56:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:35 fir-io2-s2 kernel: ll_ost00_088    D ffff90fd118f1040     0 84954      2 0x00000080
Apr 22 18:56:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:36 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:56:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:36 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:56:36 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:56:36 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:56:36 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:56:36 fir-io2-s2 kernel: LustreError: 84877:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555984296, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-fir-OST000f_UUID lock: ffff91018a600240/0x4ec9aa14f90e4b86 lrc: 3/0,1 mode: --/PW res: [0xac0000402:0x576d69:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 84877 timeout: 0 lvb_type: 0
Apr 22 18:56:37 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:37 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:37 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:56:37 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:37 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 18:56:37 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 18:56:37 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 18:56:37 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 18:56:37 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:37 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 18:56:37 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:37 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:38 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:38 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:38 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:38 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:38 fir-io2-s2 kernel: ll_ost_io01_006 D ffff90fd118f2080     0 84955      2 0x00000080
Apr 22 18:56:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:38 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 18:56:38 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 18:56:39 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:39 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:56:39 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 18:56:39 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:56:39 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:56:39 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:56:39 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:56:39 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:56:39 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:39 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:56:39 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:39 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:40 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:40 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:40 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:40 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984600.84776
Apr 22 18:56:40 fir-io2-s2 kernel: ll_ost_io00_005 D ffff90fd118f30c0     0 84956      2 0x00000080
Apr 22 18:56:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:40 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:42 fir-io2-s2 kernel: ll_ost03_056    S ffff90fd118f4100     0 84957      2 0x00000080
Apr 22 18:56:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:42 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:56:42 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:56:42 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:56:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:42 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984602.84930
Apr 22 18:56:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:43 fir-io2-s2 kernel: ll_ost00_089    D ffff90fd118f5140     0 84958      2 0x00000080
Apr 22 18:56:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:43 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:44 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:44 fir-io2-s2 kernel: ll_ost02_081    D ffff90fd118f6180     0 84959      2 0x00000080
Apr 22 18:56:44 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:44 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:56:44 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:44 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:56:44 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:56:44 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:56:44 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:56:45 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:45 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:45 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:56:45 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:45 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:56:45 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:56:45 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:56:45 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:56:45 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:56:45 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:56:45 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:46 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:46 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:46 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:46 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:46 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:46 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:46 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:46 fir-io2-s2 kernel: ll_ost_io00_006 D ffff9102876e8000     0 84960      2 0x00000080
Apr 22 18:56:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:46 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffa91d82c8>] ? zone_statistics+0x88/0xa0
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 18:56:47 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:48 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:48 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 18:56:48 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:56:48 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 18:56:48 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:56:48 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:56:48 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:56:48 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:56:48 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:56:48 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:48 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:56:48 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984608.83762
Apr 22 18:56:49 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:49 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:49 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:49 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:49 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:49 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:49 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:49 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:50 fir-io2-s2 kernel: ll_ost_io02_009 D ffff9102876e9040     0 84961      2 0x00000080
Apr 22 18:56:50 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffc16615a2>] ? lquota_disk_read+0xf2/0x390 [lquota]
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 18:56:50 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 18:56:51 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 18:56:51 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 18:56:51 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 18:56:51 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 18:56:51 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 18:56:51 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 18:56:51 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:56:51 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 18:56:51 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:56:51 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:56:51 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:56:51 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:56:52 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984611.84721
Apr 22 18:56:52 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:56:52 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:52 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:56:52 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:52 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:52 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:52 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:52 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:52 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:52 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:52 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:52 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984612.84909
Apr 22 18:56:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:53 fir-io2-s2 kernel: ll_ost03_057    S ffff9102876ea080     0 84962      2 0x00000080
Apr 22 18:56:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:53 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:56:53 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:56:53 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:56:53 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:54 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984613.84809
Apr 22 18:56:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:54 fir-io2-s2 kernel: ll_ost00_090    S ffff9102876eb0c0     0 84963      2 0x00000080
Apr 22 18:56:54 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:54 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:56:54 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:56:54 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:56:54 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:55 fir-io2-s2 kernel: ll_ost02_082    D ffff9102876ec100     0 84964      2 0x00000080
Apr 22 18:56:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:56:55 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:56:55 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:56:55 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:56:55 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:56:55 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:56:55 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:56:55 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:55 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:55 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:56:55 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:56 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:56:56 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:56:56 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:56:56 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:56:56 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:56:56 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:56:56 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:56 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:56 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:56 fir-io2-s2 kernel: Lustre: fir-OST000f: Export ffff91126541e800 already connecting from 10.9.106.36@o2ib4
Apr 22 18:56:56 fir-io2-s2 kernel: Lustre: Skipped 5 previous similar messages
Apr 22 18:56:56 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:56 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:56:57 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:56:57 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:56:57 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:56:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:56:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:56:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:56:57 fir-io2-s2 kernel: ll_ost_io01_007 D ffff9102876ed140     0 84965      2 0x00000080
Apr 22 18:56:57 fir-io2-s2 kernel: Call Trace:
Apr 22 18:56:57 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:56:57 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:56:57 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:56:57 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:56:57 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:56:58 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:56:58 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 18:56:58 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984618.84753
Apr 22 18:56:58 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 18:56:58 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 18:56:58 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 18:56:58 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 18:56:58 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 18:56:58 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:56:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:56:58 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 18:56:58 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:56:58 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 18:56:59 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:56:59 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:56:59 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984619.84852
Apr 22 18:56:59 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:56:59 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:56:59 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:56:59 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:56:59 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:56:59 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:56:59 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:56:59 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:56:59 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:56:59 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:00 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:00 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:00 fir-io2-s2 kernel: LNet: Service thread pid 84774 was inactive for 274.74s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 18:57:00 fir-io2-s2 kernel: LNet: Skipped 15 previous similar messages
Apr 22 18:57:00 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984620.84774
Apr 22 18:57:00 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:00 fir-io2-s2 kernel: ll_ost03_058    S ffff9102876ee180     0 84966      2 0x00000080
Apr 22 18:57:00 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:01 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:57:01 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:57:01 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:57:01 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:01 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:01 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:01 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:01 fir-io2-s2 kernel: ll_ost00_091    S ffff90fd11828000     0 84967      2 0x00000080
Apr 22 18:57:01 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:01 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:01 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:57:01 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:57:02 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:57:02 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:02 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:02 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984622.84958
Apr 22 18:57:02 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:02 fir-io2-s2 kernel: ll_ost03_059    D ffff90fd11829040     0 84968      2 0x00000080
Apr 22 18:57:02 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:02 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:57:02 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:57:02 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:57:02 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:57:02 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:03 fir-io2-s2 kernel: ll_ost02_083    D ffff90fd1182a080     0 84969      2 0x00000080
Apr 22 18:57:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:57:03 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:57:04 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:57:04 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:57:04 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:57:04 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984624.82943
Apr 22 18:57:04 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:57:04 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:57:04 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:04 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:04 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:04 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:04 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:04 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:05 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:05 fir-io2-s2 kernel: ll_ost_io03_005 D ffff90fd1182b0c0     0 84970      2 0x00000080
Apr 22 18:57:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:05 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:57:05 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:57:05 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:57:05 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:57:05 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:57:05 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:06 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:07 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:07 fir-io2-s2 kernel: ll_ost_io01_008 D ffff90fd1182c100     0 84971      2 0x00000080
Apr 22 18:57:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:07 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 18:57:07 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 18:57:07 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:57:07 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:08 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:09 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:09 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:09 fir-io2-s2 kernel: ll_ost_io00_007 D ffff90fd1182d140     0 84972      2 0x00000080
Apr 22 18:57:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:09 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:57:09 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:57:09 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:57:09 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:57:09 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:57:10 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:57:10 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 18:57:10 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 18:57:10 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 18:57:10 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 18:57:10 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 18:57:10 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 18:57:10 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 18:57:10 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 18:57:10 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 18:57:10 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 18:57:10 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 18:57:11 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:57:11 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 18:57:11 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:57:11 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:57:11 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:57:11 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:57:11 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:57:11 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:57:11 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:57:11 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:11 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:11 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:12 fir-io2-s2 kernel: ll_ost03_060    S ffff90fd1182e180     0 84973      2 0x00000080
Apr 22 18:57:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:57:12 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:57:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:13 fir-io2-s2 kernel: ll_ost00_092    D ffff910216520000     0 84974      2 0x00000080
Apr 22 18:57:13 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:13 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984633.83761
Apr 22 18:57:13 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:57:13 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:57:13 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:57:13 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:57:13 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:57:13 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:14 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:14 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:14 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:14 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:14 fir-io2-s2 kernel: ll_ost_io03_006 D ffff910216521040     0 84975      2 0x00000080
Apr 22 18:57:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:57:15 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:57:16 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:57:16 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:57:16 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:57:16 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:57:16 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:16 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:16 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984636.82478
Apr 22 18:57:16 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:16 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:16 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:16 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:17 fir-io2-s2 kernel: Lustre: fir-OST0013: haven't heard from client 95d140f9-df98-0fce-448c-5b0fa3c607d4 (at 10.9.106.28@o2ib4) in 397 seconds. I think it's dead, and I am evicting it. exp ffff9112612b3000, cur 1555984637 expire 1555984487 last 1555984240
Apr 22 18:57:17 fir-io2-s2 kernel: Lustre: Skipped 18 previous similar messages
Apr 22 18:57:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:17 fir-io2-s2 kernel: ll_ost_io02_010 D ffff910216522080     0 84976      2 0x00000080
Apr 22 18:57:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:17 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:57:17 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:57:18 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:57:18 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:57:18 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:57:18 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:57:18 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:57:18 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:18 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:57:18 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984638.84805
Apr 22 18:57:18 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:18 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:57:18 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 18:57:18 fir-io2-s2 kernel: LustreError: 83732:0:(ldlm_request.c:129:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1555984338, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-fir-OST0015_UUID lock: ffff90e4a9628b40/0x4ec9aa14f922af25 lrc: 3/0,1 mode: --/PW res: [0x740000402:0x929092:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 83732 timeout: 0 lvb_type: 0
Apr 22 18:57:19 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 18:57:19 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:19 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:19 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:19 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:19 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:19 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:20 fir-io2-s2 kernel: ll_ost00_093    S ffff9102165230c0     0 84977      2 0x00000080
Apr 22 18:57:20 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:21 fir-io2-s2 kernel: ll_ost03_061    S ffff910216524100     0 84978      2 0x00000080
Apr 22 18:57:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:21 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:57:21 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:57:21 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:57:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:21 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984641.83754
Apr 22 18:57:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:22 fir-io2-s2 kernel: ll_ost02_084    D ffff910216525140     0 84979      2 0x00000080
Apr 22 18:57:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:22 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:57:22 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:57:22 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:57:22 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:57:22 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:57:22 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:57:22 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:57:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:22 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:57:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:23 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:57:23 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:57:23 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:57:23 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:57:23 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:57:23 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:57:23 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:23 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:23 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:23 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:23 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:23 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:24 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:24 fir-io2-s2 kernel: ll_ost_io02_011 D ffff910216526180     0 84980      2 0x00000080
Apr 22 18:57:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:24 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:57:24 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:57:24 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:57:24 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984644.84824
Apr 22 18:57:24 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:57:24 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:57:25 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:57:25 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 18:57:25 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 18:57:25 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 18:57:25 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 18:57:25 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 18:57:25 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 18:57:25 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 18:57:25 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 18:57:25 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 18:57:25 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 18:57:25 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:57:26 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 18:57:26 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:57:26 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:57:26 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:57:26 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:57:26 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:57:26 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:57:26 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:57:26 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:26 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:26 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:26 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:27 fir-io2-s2 kernel: ll_ost_io00_008 D ffff910216598000     0 84981      2 0x00000080
Apr 22 18:57:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:57:27 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:57:28 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:57:28 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:57:28 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:57:28 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:28 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:57:28 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:28 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:57:28 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 18:57:28 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 18:57:28 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:28 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:28 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:29 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:29 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:29 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:29 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:29 fir-io2-s2 kernel: ll_ost03_062    S ffff910216599040     0 84982      2 0x00000080
Apr 22 18:57:29 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:29 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:57:29 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984649.82385
Apr 22 18:57:30 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:57:30 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:57:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:30 fir-io2-s2 kernel: ll_ost00_094    S ffff91021659a080     0 84983      2 0x00000080
Apr 22 18:57:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:30 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:57:30 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:57:30 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:57:30 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984650.83740
Apr 22 18:57:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:31 fir-io2-s2 kernel: ll_ost02_085    D ffff91021659b0c0     0 84984      2 0x00000080
Apr 22 18:57:31 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:31 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:57:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:57:31 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:57:31 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:57:31 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:57:31 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:57:32 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:57:32 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:32 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:57:32 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:32 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:57:32 fir-io2-s2 kernel: [<ffffffffc1807d83>] ofd_attr_set+0x473/0xb50 [ofd]
Apr 22 18:57:32 fir-io2-s2 kernel: [<ffffffffc17fa71d>] ofd_setattr_hdl+0x31d/0x8f0 [ofd]
Apr 22 18:57:32 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:32 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:32 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:32 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:32 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:33 fir-io2-s2 kernel: ll_ost03_063    D ffff91021659c100     0 84985      2 0x00000080
Apr 22 18:57:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:57:33 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:34 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:34 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:34 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:34 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:34 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:34 fir-io2-s2 kernel: ll_ost_io01_009 D ffff91021659d140     0 84986      2 0x00000080
Apr 22 18:57:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:34 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:57:35 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:57:35 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:57:35 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:57:35 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:57:35 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:57:35 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 18:57:35 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 18:57:35 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 18:57:35 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 18:57:35 fir-io2-s2 kernel: [<ffffffffc16615a2>] ? lquota_disk_read+0xf2/0x390 [lquota]
Apr 22 18:57:35 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 18:57:35 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:57:36 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:57:37 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:57:37 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:37 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:37 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:37 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:37 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:37 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:37 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:38 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984658.84822
Apr 22 18:57:38 fir-io2-s2 kernel: ll_ost00_095    D ffff91021659e180     0 84987      2 0x00000080
Apr 22 18:57:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:38 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:57:38 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:57:38 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:57:38 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:57:38 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:57:38 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:57:38 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:57:38 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:38 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:39 fir-io2-s2 kernel: ll_ost02_086    D ffff910217210000     0 84988      2 0x00000080
Apr 22 18:57:39 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:57:39 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:57:40 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:57:40 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:57:40 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:57:40 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:40 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:40 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:40 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:41 fir-io2-s2 kernel: ll_ost03_064    D ffff910217211040     0 84989      2 0x00000080
Apr 22 18:57:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:57:41 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:42 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 18:57:42 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 18:57:42 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 18:57:42 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 18:57:42 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 18:57:42 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:42 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:42 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:42 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:42 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:42 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:43 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984663.84841
Apr 22 18:57:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:43 fir-io2-s2 kernel: ll_ost_io02_012 D ffff910217212080     0 84990      2 0x00000080
Apr 22 18:57:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:43 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:57:43 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:57:43 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:57:43 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:57:43 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:57:43 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:57:43 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:57:44 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:45 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:46 fir-io2-s2 kernel: ll_ost00_096    S ffff9102172130c0     0 84991      2 0x00000080
Apr 22 18:57:46 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:46 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:57:46 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:57:46 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:57:46 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:47 fir-io2-s2 kernel: ll_ost_io03_007 D ffff910217214100     0 84992      2 0x00000080
Apr 22 18:57:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:47 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:57:47 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:57:47 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:57:47 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:57:47 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:57:47 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:57:47 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:57:47 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:48 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:49 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:49 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:49 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:49 fir-io2-s2 kernel: ll_ost02_087    D ffff910217215140     0 84993      2 0x00000080
Apr 22 18:57:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:49 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:57:49 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:57:49 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:57:49 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:57:50 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:57:50 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:57:50 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:57:50 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:50 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984670.84817
Apr 22 18:57:50 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:57:50 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:50 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:57:50 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:57:50 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:57:50 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:57:51 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:57:51 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:57:51 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:51 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:51 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:51 fir-io2-s2 kernel: Lustre: 84927:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff90fed76b6c50 x1631475203394512/t0(0) o6->fir-MDT0002-mdtlov_UUID@10.0.10.51@o2ib7:646/0 lens 544/432 e 17 to 0 dl 1555984676 ref 2 fl Interpret:/0/0 rc 0/0
Apr 22 18:57:51 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:51 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:51 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:51 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:52 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:52 fir-io2-s2 kernel: ll_ost_io01_010 D ffff910217216180     0 84994      2 0x00000080
Apr 22 18:57:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:52 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:57:52 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:57:52 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:57:52 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:57:52 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:57:52 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:57:53 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:57:54 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:54 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:54 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:54 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:54 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:54 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:54 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:54 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:55 fir-io2-s2 kernel: ll_ost_io02_013 D ffff9102172a0000     0 84995      2 0x00000080
Apr 22 18:57:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffa92ae6cc>] ? dqput+0x4c/0x1f0
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffc16615a2>] ? lquota_disk_read+0xf2/0x390 [lquota]
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 18:57:55 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 18:57:56 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 18:57:56 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 18:57:56 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 18:57:56 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 18:57:56 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 18:57:56 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 18:57:56 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:57:56 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 18:57:56 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:57:56 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:57:56 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:57:56 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:57:57 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:57:57 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:57:57 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:57:57 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:57:57 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:57:57 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:57:57 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:57:57 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:57:57 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:57:57 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:57:57 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:58 fir-io2-s2 kernel: ll_ost03_065    S ffff9102172a1040     0 84996      2 0x00000080
Apr 22 18:57:58 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:58 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:57:58 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:57:58 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:57:58 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:59 fir-io2-s2 kernel: ll_ost00_097    S ffff9102172a2080     0 84997      2 0x00000080
Apr 22 18:57:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:57:59 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:57:59 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:57:59 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:57:59 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:57:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:57:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:59 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984679.84764
Apr 22 18:57:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:57:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:57:59 fir-io2-s2 kernel: ll_ost_io03_008 D ffff9102172a30c0     0 84998      2 0x00000080
Apr 22 18:57:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:57:59 fir-io2-s2 kernel: [<ffffffffc0c7f56a>] ? lnet_me_unlink+0x5a/0xc0 [lnet]
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa90cec64>] ? __wake_up+0x44/0x50
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa9765b71>] out_of_line_wait_on_bit_lock+0x81/0xb0
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:58:00 fir-io2-s2 kernel: [<ffffffffa9278f42>] __lock_buffer+0x32/0x40
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffc09e1ea2>] do_get_write_access+0x432/0x4d0 [jbd2]
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffc15b0dc0>] ldiskfs_reserve_inode_write+0x70/0xa0 [ldiskfs]
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ? ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffc15b0e43>] ldiskfs_mark_inode_dirty+0x53/0x210 [ldiskfs]
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffa927012d>] __mark_inode_dirty+0x16d/0x270
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffc15e59fd>] ldiskfs_mb_new_blocks+0x19d/0xb20 [ldiskfs]
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 18:58:01 fir-io2-s2 kernel: [<ffffffffc16763c3>] ? qsd_acquire+0x8e3/0xcb0 [lquota]
Apr 22 18:58:02 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 18:58:02 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 18:58:02 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 18:58:02 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 18:58:02 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:58:02 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 18:58:02 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:02 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:02 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:02 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:02 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:02 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:03 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:03 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:03 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:03 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:03 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:03 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:03 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:03 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:03 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984683.84748
Apr 22 18:58:03 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:04 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:04 fir-io2-s2 kernel: ll_ost_io01_011 D ffff9102172a4100     0 84999      2 0x00000080
Apr 22 18:58:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:04 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:58:04 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:04 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:58:04 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:58:04 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:58:04 fir-io2-s2 kernel: LNet: Service thread pid 84944 was inactive for 258.38s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 18:58:04 fir-io2-s2 kernel: LNet: Skipped 57 previous similar messages
Apr 22 18:58:04 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984684.84944
Apr 22 18:58:05 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:58:05 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:58:05 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:05 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:58:05 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:05 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:58:05 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:58:05 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:05 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:05 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:05 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:05 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:07 fir-io2-s2 kernel: ll_ost03_066    S ffff9102172a5140     0 85000      2 0x00000080
Apr 22 18:58:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:07 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:58:07 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:58:07 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:58:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:07 fir-io2-s2 kernel: ll_ost00_098    D ffff9102172a6180     0 85001      2 0x00000080
Apr 22 18:58:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:07 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:09 fir-io2-s2 kernel: ll_ost_io00_009 D ffff91021f908000     0 85002      2 0x00000080
Apr 22 18:58:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:09 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:58:09 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:09 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:58:09 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:58:09 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:58:09 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:58:09 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:58:09 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:09 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:10 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:11 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:11 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:11 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:11 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:11 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:11 fir-io2-s2 kernel: ll_ost_io02_014 D ffff91021f909040     0 85003      2 0x00000080
Apr 22 18:58:11 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:11 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:58:11 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:12 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:58:12 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:58:12 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:58:12 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:58:12 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:58:12 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:12 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:58:12 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:12 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:58:12 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:12 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:12 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:13 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:13 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:13 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:13 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:13 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:13 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:13 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:13 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:13 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:13 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:13 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:13 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:14 fir-io2-s2 kernel: ll_ost_io01_012 D ffff91021f90a080     0 85004      2 0x00000080
Apr 22 18:58:14 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:14 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:15 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:15 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:15 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:15 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:15 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:15 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:15 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:15 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:15 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:16 fir-io2-s2 kernel: ll_ost02_088    D ffff91021f90b0c0     0 85005      2 0x00000080
Apr 22 18:58:16 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:16 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:17 fir-io2-s2 kernel: ldlm_cn02_004   S ffff91021f90c100     0 85006      2 0x00000080
Apr 22 18:58:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:17 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:58:17 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:58:18 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:58:18 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:18 fir-io2-s2 kernel: ldlm_cn03_003   S ffff91021f90d140     0 85007      2 0x00000080
Apr 22 18:58:18 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:18 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:58:18 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:58:18 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:58:18 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:19 fir-io2-s2 kernel: ll_ost_io00_010 D ffff91021f90e180     0 85008      2 0x00000080
Apr 22 18:58:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:19 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:58:19 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:19 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:58:19 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:58:19 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:58:19 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:58:19 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:58:20 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:20 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:58:20 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:20 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:58:20 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:58:20 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:20 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:20 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:20 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:20 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:20 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:20 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:21 fir-io2-s2 kernel: ll_ost_io01_013 D ffff91021f988000     0 85009      2 0x00000080
Apr 22 18:58:21 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:22 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:58:22 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:22 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:58:22 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:22 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:58:22 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:58:22 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:58:22 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:58:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:22 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:58:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:22 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:58:23 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:23 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:23 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:23 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:23 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:23 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:23 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:23 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:23 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:23 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:23 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:23 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:24 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:24 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:24 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984704.83765
Apr 22 18:58:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:24 fir-io2-s2 kernel: ll_ost00_099    S ffff91021f989040     0 85010      2 0x00000080
Apr 22 18:58:24 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:24 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:58:24 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:58:24 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:58:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:25 fir-io2-s2 kernel: ll_ost02_089    D ffff91021f98a080     0 85011      2 0x00000080
Apr 22 18:58:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:25 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:58:25 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:25 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:58:25 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:58:25 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:58:25 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:58:25 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:58:26 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:26 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:58:26 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:26 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:58:26 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:58:26 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:58:26 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:58:26 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:58:26 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:58:26 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:26 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:27 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:27 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:27 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:27 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984707.85029
Apr 22 18:58:27 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:27 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:27 fir-io2-s2 kernel: ll_ost_io02_015 D ffff91021f98b0c0     0 85012      2 0x00000080
Apr 22 18:58:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:58:28 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984708.85403
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:28 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:29 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:29 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:29 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:29 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984709.85047
Apr 22 18:58:29 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:29 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:29 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:29 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:29 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:29 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:29 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:58:29 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:30 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984710.82413
Apr 22 18:58:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:30 fir-io2-s2 kernel: ll_ost_io01_014 D ffff91021f98c100     0 85013      2 0x00000080
Apr 22 18:58:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:30 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 18:58:30 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 18:58:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:30 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:58:30 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 18:58:31 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:31 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:31 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:31 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:31 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984711.84970
Apr 22 18:58:31 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:31 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:31 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:31 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:31 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:31 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:31 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:31 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:32 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:32 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:32 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984712.85402
Apr 22 18:58:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:32 fir-io2-s2 kernel: ll_ost03_067    S ffff91021f98d140     0 85014      2 0x00000080
Apr 22 18:58:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:32 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:58:32 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:58:33 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:58:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:33 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984713.85339
Apr 22 18:58:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:33 fir-io2-s2 kernel: ll_ost03_068    S ffff91021f98e180     0 85015      2 0x00000080
Apr 22 18:58:33 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:33 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:58:33 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:58:33 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:58:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:34 fir-io2-s2 kernel: ll_ost_io00_011 D ffff910220e10000     0 85016      2 0x00000080
Apr 22 18:58:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:34 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984714.84765
Apr 22 18:58:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:34 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:58:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:34 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:58:34 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:58:34 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:58:34 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:58:34 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:58:35 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:35 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:58:35 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:35 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:58:35 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984715.85285
Apr 22 18:58:35 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:35 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:35 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:35 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:35 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:35 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:36 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:36 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:36 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:36 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:36 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:36 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:36 fir-io2-s2 kernel: Lustre: 84927:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff90fe4d027c50 x1631475203826608/t0(0) o6->fir-MDT0002-mdtlov_UUID@10.0.10.51@o2ib7:691/0 lens 544/432 e 9 to 0 dl 1555984721 ref 2 fl Interpret:/0/0 rc 0/0
Apr 22 18:58:36 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:36 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:36 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:37 fir-io2-s2 kernel: ll_ost02_090    D ffff910220e11040     0 85017      2 0x00000080
Apr 22 18:58:37 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:37 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:58:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:37 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:58:37 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984717.85232
Apr 22 18:58:37 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:58:37 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:58:37 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:58:37 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:58:37 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:58:37 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:58:37 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:58:37 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:58:38 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:58:38 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:58:38 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 18:58:38 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:58:38 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:58:38 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984718.85327
Apr 22 18:58:38 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 18:58:38 fir-io2-s2 kernel: [<ffffffffa927882d>] ? __brelse+0x3d/0x50
Apr 22 18:58:38 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 18:58:38 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 18:58:38 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 18:58:38 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 18:58:39 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 18:58:39 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 18:58:39 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 18:58:39 fir-io2-s2 kernel: [<ffffffffc16d8f00>] ? osd_oi_insert+0x4d0/0x520 [osd_ldiskfs]
Apr 22 18:58:39 fir-io2-s2 kernel: [<ffffffffc16c7edf>] ? osd_destroy+0x1af/0x760 [osd_ldiskfs]
Apr 22 18:58:39 fir-io2-s2 kernel: [<ffffffffc16c723d>] ? osd_ref_del+0x1ad/0x6a0 [osd_ldiskfs]
Apr 22 18:58:39 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984719.85198
Apr 22 18:58:39 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:58:39 fir-io2-s2 kernel: [<ffffffffc1809455>] ofd_destroy+0x2c5/0x960 [ofd]
Apr 22 18:58:39 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:58:39 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:58:39 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:58:39 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:58:40 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:40 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:40 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:40 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:40 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984720.85261
Apr 22 18:58:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:41 fir-io2-s2 kernel: ll_ost_io02_016 D ffff910220e12080     0 85018      2 0x00000080
Apr 22 18:58:41 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:41 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:58:41 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:41 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:58:41 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:58:41 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:58:41 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984721.85337
Apr 22 18:58:41 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:58:41 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:58:41 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:41 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:42 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:43 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:43 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:43 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:43 fir-io2-s2 kernel: ll_ost_io01_015 D ffff910220e130c0     0 85019      2 0x00000080
Apr 22 18:58:43 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984723.85414
Apr 22 18:58:43 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:43 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:43 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 18:58:43 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 18:58:43 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:43 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:58:44 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 18:58:44 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:44 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:44 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:44 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:44 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:44 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:44 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:44 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:44 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:44 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:44 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:45 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:45 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:45 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:45 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:45 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984725.47272
Apr 22 18:58:45 fir-io2-s2 kernel: ll_ost_io03_009 D ffff910220e14100     0 85020      2 0x00000080
Apr 22 18:58:45 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:45 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:45 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:45 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:58:46 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:58:46 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:58:46 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:58:46 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:58:46 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:46 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:46 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:46 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:46 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:46 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:46 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:47 fir-io2-s2 kernel: ll_ost03_069    S ffff910220e15140     0 85021      2 0x00000080
Apr 22 18:58:47 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:47 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:58:47 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:58:47 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:58:47 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:47 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984727.47304
Apr 22 18:58:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:48 fir-io2-s2 kernel: ll_ost_io00_012 D ffff910220e16180     0 85022      2 0x00000080
Apr 22 18:58:48 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:48 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:58:48 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:58:48 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:58:48 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:58:48 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:58:48 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:58:48 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 18:58:48 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 18:58:48 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984728.47335
Apr 22 18:58:48 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 18:58:48 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 18:58:49 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 18:58:49 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 18:58:49 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 18:58:49 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:58:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:49 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 18:58:49 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:58:49 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 18:58:49 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:49 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:49 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984729.47348
Apr 22 18:58:49 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:49 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:50 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:50 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:50 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:50 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:50 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:50 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:50 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:50 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:50 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:50 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:50 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984730.47316
Apr 22 18:58:50 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:51 fir-io2-s2 kernel: ll_ost_io00_013 D ffff910220e38000     0 85023      2 0x00000080
Apr 22 18:58:51 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:51 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:58:51 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:58:51 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:58:51 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:58:51 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:58:51 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984731.47387
Apr 22 18:58:51 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:58:51 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:58:52 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:52 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:58:52 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:52 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:58:52 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 18:58:52 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 18:58:52 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:52 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:52 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:52 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:52 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:52 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:53 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:53 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:53 fir-io2-s2 kernel: ll_ost_io02_017 D ffff910220e39040     0 85024      2 0x00000080
Apr 22 18:58:53 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:53 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:58:53 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:58:53 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:58:53 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:58:53 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984733.47419
Apr 22 18:58:53 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:58:53 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:58:54 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 18:58:54 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 18:58:54 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 18:58:54 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 18:58:54 fir-io2-s2 kernel: [<ffffffffc16615a2>] ? lquota_disk_read+0xf2/0x390 [lquota]
Apr 22 18:58:54 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 18:58:54 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 18:58:54 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 18:58:54 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 18:58:54 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 18:58:54 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 18:58:54 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 18:58:55 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 18:58:55 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:58:55 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 18:58:55 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:55 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:55 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:55 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:55 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:58:55 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:58:55 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:55 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:55 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:56 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:56 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:56 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:56 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:56 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:56 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:56 fir-io2-s2 kernel: ll_ost_io00_014 D ffff910220e3a080     0 85025      2 0x00000080
Apr 22 18:58:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:56 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:58:57 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:58:57 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984737.84781
Apr 22 18:58:58 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 18:58:58 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:58:58 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 18:58:58 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:58:58 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:58:58 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:58:58 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:58:58 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:58:58 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:58:58 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:58:58 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:58:58 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:58:59 fir-io2-s2 kernel: ll_ost02_091    D ffff910220e3b0c0     0 85026      2 0x00000080
Apr 22 18:58:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:58:59 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffa927882d>] ? __brelse+0x3d/0x50
Apr 22 18:59:00 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 18:59:01 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 18:59:01 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 18:59:01 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 18:59:01 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 18:59:01 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 18:59:01 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 18:59:01 fir-io2-s2 kernel: [<ffffffffc16d8f00>] ? osd_oi_insert+0x4d0/0x520 [osd_ldiskfs]
Apr 22 18:59:01 fir-io2-s2 kernel: [<ffffffffc16c7edf>] ? osd_destroy+0x1af/0x760 [osd_ldiskfs]
Apr 22 18:59:01 fir-io2-s2 kernel: [<ffffffffc16c723d>] ? osd_ref_del+0x1ad/0x6a0 [osd_ldiskfs]
Apr 22 18:59:01 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:59:01 fir-io2-s2 kernel: [<ffffffffc1809455>] ofd_destroy+0x2c5/0x960 [ofd]
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:02 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:03 fir-io2-s2 kernel: ll_ost02_092    D ffff910220e3c100     0 85027      2 0x00000080
Apr 22 18:59:03 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:03 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:59:03 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:59:03 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:59:03 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:59:03 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:59:03 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:03 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:03 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:03 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:04 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:04 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:04 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:04 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:04 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:04 fir-io2-s2 kernel: ll_ost03_070    S ffff910220e3d140     0 85028      2 0x00000080
Apr 22 18:59:04 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:04 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:59:04 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:59:04 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:05 fir-io2-s2 kernel: ll_ost_io03_010 D ffff910220e3e180     0 85029      2 0x00000080
Apr 22 18:59:05 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:59:05 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:59:06 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:59:06 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:59:06 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984746.85182
Apr 22 18:59:06 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:06 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 18:59:06 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:06 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:06 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:06 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:06 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:06 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:06 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:07 fir-io2-s2 kernel: ll_ost02_093    D ffff910224800000     0 85030      2 0x00000080
Apr 22 18:59:07 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:07 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:59:07 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:59:07 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:59:07 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:59:07 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:59:07 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:59:08 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:08 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:59:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:08 fir-io2-s2 kernel: [<ffffffffc1805f17>] ofd_precreate_objects+0xa57/0x1ab0 [ofd]
Apr 22 18:59:08 fir-io2-s2 kernel: [<ffffffffc17f8794>] ofd_create_hdl+0x474/0x20e0 [ofd]
Apr 22 18:59:08 fir-io2-s2 kernel: [<ffffffffc123f4b7>] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc]
Apr 22 18:59:08 fir-io2-s2 kernel: [<ffffffffc123f80f>] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc]
Apr 22 18:59:08 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:08 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:08 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:09 fir-io2-s2 kernel: ll_ost_io03_011 D ffff910224801040     0 85031      2 0x00000080
Apr 22 18:59:09 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:59:09 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:59:10 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:59:10 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:59:10 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:10 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984750.85088
Apr 22 18:59:10 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:10 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:59:10 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:10 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:59:10 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:59:10 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:59:10 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:59:10 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:12 fir-io2-s2 kernel: ll_ost_io01_016 D ffff910224802080     0 85032      2 0x00000080
Apr 22 18:59:12 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 18:59:12 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 18:59:13 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 18:59:13 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 18:59:13 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:13 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:13 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 18:59:13 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:59:13 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 18:59:13 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:59:13 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:59:13 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:59:13 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:59:13 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:14 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:59:14 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:59:14 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:14 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:14 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:14 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:14 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:14 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:15 fir-io2-s2 kernel: ll_ost_io03_012 D ffff9102248030c0     0 85033      2 0x00000080
Apr 22 18:59:15 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:59:15 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:16 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 18:59:16 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:16 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:16 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:16 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:16 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:16 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:16 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:17 fir-io2-s2 kernel: ll_ost_io03_013 D ffff910224804100     0 85034      2 0x00000080
Apr 22 18:59:17 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:17 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:59:17 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:59:17 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:59:17 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:59:17 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:59:17 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:59:17 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:17 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:17 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:59:17 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:18 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:59:18 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:59:18 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:59:18 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:59:18 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:59:18 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:18 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:59:18 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:59:18 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:18 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:19 fir-io2-s2 kernel: ll_ost02_094    D ffff910224805140     0 85035      2 0x00000080
Apr 22 18:59:19 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:59:19 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:59:20 fir-io2-s2 kernel: [<ffffffffc15b0dc0>] ldiskfs_reserve_inode_write+0x70/0xa0 [ldiskfs]
Apr 22 18:59:21 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:21 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ? ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 18:59:21 fir-io2-s2 kernel: [<ffffffffc15b0e43>] ldiskfs_mark_inode_dirty+0x53/0x210 [ldiskfs]
Apr 22 18:59:21 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 18:59:21 fir-io2-s2 kernel: [<ffffffffc16c722d>] osd_ref_del+0x19d/0x6a0 [osd_ldiskfs]
Apr 22 18:59:21 fir-io2-s2 kernel: [<ffffffffc1809699>] ofd_destroy+0x509/0x960 [ofd]
Apr 22 18:59:21 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:59:21 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:59:21 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:59:21 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:59:21 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:21 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:22 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:22 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:22 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:22 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:22 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:22 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:22 fir-io2-s2 kernel: ll_ost03_071    S ffff910224806180     0 85036      2 0x00000080
Apr 22 18:59:22 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:23 fir-io2-s2 kernel: ll_ost02_095    D ffff910224840000     0 85037      2 0x00000080
Apr 22 18:59:23 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:59:23 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:59:24 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:59:24 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:59:24 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:24 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:24 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:24 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:24 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:24 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:25 fir-io2-s2 kernel: ll_ost02_096    D ffff910224841040     0 85038      2 0x00000080
Apr 22 18:59:25 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:25 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:59:25 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:59:25 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:59:25 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:59:25 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:59:25 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:59:25 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:25 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:25 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:59:25 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:26 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:59:26 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:59:26 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:59:26 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:59:26 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:59:26 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:59:26 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:26 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:26 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:26 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:26 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:59:26 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:27 fir-io2-s2 kernel: Lustre: fir-OST0017: Export ffff90f1c8734800 already connecting from 10.9.106.36@o2ib4
Apr 22 18:59:27 fir-io2-s2 kernel: Lustre: Skipped 13 previous similar messages
Apr 22 18:59:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:27 fir-io2-s2 kernel: ll_ost_io00_015 D ffff910224842080     0 85039      2 0x00000080
Apr 22 18:59:27 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:27 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:59:27 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:59:27 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:59:27 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:28 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:59:29 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:59:29 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:29 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:29 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:29 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:29 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:29 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:29 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:30 fir-io2-s2 kernel: ll_ost02_097    D ffff9102248430c0     0 85040      2 0x00000080
Apr 22 18:59:30 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:30 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:59:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:59:30 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:59:30 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:59:30 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:59:30 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:59:30 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:30 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:30 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:59:30 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:31 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:59:31 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:59:31 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:59:31 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:59:31 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:59:31 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:59:31 fir-io2-s2 kernel: Lustre: 84927:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff90fbbb799c50 x1631475204294256/t0(0) o6->fir-MDT0002-mdtlov_UUID@10.0.10.51@o2ib7:746/0 lens 544/432 e 5 to 0 dl 1555984776 ref 2 fl Interpret:/0/0 rc 0/0
Apr 22 18:59:31 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:31 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:32 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:32 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:32 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:32 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:32 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:32 fir-io2-s2 kernel: ll_ost02_098    D ffff910224844100     0 85041      2 0x00000080
Apr 22 18:59:32 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:32 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:34 fir-io2-s2 kernel: ll_ost03_072    S ffff910224845140     0 85042      2 0x00000080
Apr 22 18:59:34 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:34 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:34 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:59:34 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:59:34 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:59:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:35 fir-io2-s2 kernel: ll_ost_io00_016 D ffff910224846180     0 85043      2 0x00000080
Apr 22 18:59:35 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:35 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:59:35 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:59:35 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:59:35 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:59:35 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:59:35 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:59:35 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 18:59:35 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 18:59:35 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 18:59:35 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 18:59:35 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 18:59:36 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 18:59:36 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:36 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 18:59:36 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:59:36 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 18:59:36 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:59:36 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:59:36 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:59:36 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:59:36 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:36 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:38 fir-io2-s2 kernel: ll_ost02_099    D ffff910224890000     0 85044      2 0x00000080
Apr 22 18:59:38 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:38 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:59:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:59:38 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:59:38 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:59:38 fir-io2-s2 kernel: [<ffffffffc16e6740>] ? osd_declare_write+0x330/0x470 [osd_ldiskfs]
Apr 22 18:59:38 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:59:38 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:38 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:38 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:59:38 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:38 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984778.84969
Apr 22 18:59:39 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:59:39 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:59:39 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:59:39 fir-io2-s2 kernel: [<ffffffffc120f660>] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc]
Apr 22 18:59:39 fir-io2-s2 kernel: [<ffffffffc1209940>] ? ldlm_expired_completion_wait+0x220/0x220 [ptlrpc]
Apr 22 18:59:39 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:59:39 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:39 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:39 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:39 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:39 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:40 fir-io2-s2 kernel: ll_ost_io00_017 D ffff910224891040     0 85045      2 0x00000080
Apr 22 18:59:40 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:40 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:41 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:59:41 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:41 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:59:41 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 18:59:41 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:59:41 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:59:41 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:59:41 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:59:41 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:41 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:59:41 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:59:41 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:42 fir-io2-s2 kernel: ll_ost_io02_018 D ffff910224892080     0 85046      2 0x00000080
Apr 22 18:59:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:42 fir-io2-s2 kernel: LNet: Service thread pid 84853 was inactive for 486.52s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Apr 22 18:59:42 fir-io2-s2 kernel: Pid: 84853, comm: ll_ost02_051 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 18:59:42 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:42 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 18:59:42 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984782.84853
Apr 22 18:59:49 fir-io2-s2 kernel: Pid: 84849, comm: ll_ost02_049 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 18:59:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 18:59:49 fir-io2-s2 kernel: Pid: 84744, comm: ll_ost02_024 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 18:59:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 18:59:49 fir-io2-s2 kernel: Pid: 84877, comm: ll_ost02_057 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 18:59:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc1209e25>] ldlm_completion_ast+0x4e5/0x890 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc120ac8c>] ldlm_cli_enqueue_local+0x23c/0x870 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc18016ad>] ofd_destroy_by_fid+0x1dd/0x4a0 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 18:59:49 fir-io2-s2 kernel: Pid: 82583, comm: ll_ost02_006 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 18:59:49 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc1809760>] ofd_destroy+0x5d0/0x960 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc18016c4>] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc17f64f7>] ofd_destroy_hdl+0x267/0x970 [ofd]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 18:59:49 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984787.84718
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 18:59:49 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:59:50 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 18:59:50 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 18:59:50 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984790.85027
Apr 22 18:59:50 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 18:59:50 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 18:59:50 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 18:59:50 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 18:59:50 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 18:59:50 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 18:59:50 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 18:59:50 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 18:59:50 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 18:59:50 fir-io2-s2 kernel: [<ffffffffc15a1fc9>] ldiskfs_orphan_add+0xb9/0x240 [ldiskfs]
Apr 22 18:59:51 fir-io2-s2 kernel: [<ffffffffc16e759c>] osd_punch+0x28c/0x6b0 [osd_ldiskfs]
Apr 22 18:59:51 fir-io2-s2 kernel: [<ffffffffc1808c4b>] ofd_object_punch+0x7eb/0xd30 [ofd]
Apr 22 18:59:51 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984791.84837
Apr 22 18:59:51 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 18:59:51 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:51 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:51 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:51 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:51 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:51 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:51 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:51 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:52 fir-io2-s2 kernel: ll_ost_io03_014 D ffff9102248930c0     0 85047      2 0x00000080
Apr 22 18:59:52 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:52 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:59:52 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:59:52 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:59:52 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:59:52 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:59:52 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:59:52 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 18:59:52 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 18:59:52 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 18:59:53 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 18:59:53 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 18:59:53 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 18:59:53 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 18:59:53 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 18:59:53 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 18:59:53 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 18:59:53 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:59:53 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 18:59:53 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:59:53 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:59:53 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:59:54 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:59:54 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:54 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:59:54 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984794.82416
Apr 22 18:59:54 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:59:54 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:54 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:54 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:54 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:54 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:54 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:54 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:54 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:55 fir-io2-s2 kernel: ll_ost_io00_018 S ffff910224894100     0 85048      2 0x00000080
Apr 22 18:59:55 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:55 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:59:55 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:59:55 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:59:55 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:56 fir-io2-s2 kernel: ll_ost_io01_017 D ffff910224895140     0 85049      2 0x00000080
Apr 22 18:59:56 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 18:59:56 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 18:59:57 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 18:59:57 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 18:59:57 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 18:59:57 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 18:59:57 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 18:59:57 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 18:59:57 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 18:59:57 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 18:59:57 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 18:59:57 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 18:59:57 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 18:59:57 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 18:59:58 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 18:59:58 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 18:59:58 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 18:59:58 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 18:59:58 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 18:59:58 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 18:59:58 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 18:59:58 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 18:59:58 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 18:59:58 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:59 fir-io2-s2 kernel: ll_ost03_073    S ffff910224896180     0 85050      2 0x00000080
Apr 22 18:59:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 18:59:59 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 18:59:59 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 18:59:59 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 18:59:59 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 18:59:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 18:59:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 18:59:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 18:59:59 fir-io2-s2 kernel: ll_ost_io02_019 D ffff90fec2020000     0 85051      2 0x00000080
Apr 22 18:59:59 fir-io2-s2 kernel: Call Trace:
Apr 22 18:59:59 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:00:00 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:00:00 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:00:00 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:00:00 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:00:00 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:00:00 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:00:00 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:00:00 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:00:00 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:00:00 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:00:00 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:00:00 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:01 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:01 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:00:01 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:00:01 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:00:01 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:01 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:01 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984801.82386
Apr 22 19:00:01 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:01 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:01 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:01 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:01 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:01 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:02 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:02 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:02 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:02 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:02 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:02 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:02 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:02 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:02 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:02 fir-io2-s2 kernel: ll_ost_io00_019 D ffff90fec2021040     0 85052      2 0x00000080
Apr 22 19:00:02 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:03 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:00:03 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:00:03 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:00:03 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:00:03 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:00:03 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:00:03 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:03 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:03 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:00:03 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:03 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:00:03 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:04 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:05 fir-io2-s2 kernel: ll_ost_io01_018 D ffff90fec2022080     0 85053      2 0x00000080
Apr 22 19:00:05 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:05 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:00:05 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:00:05 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:00:05 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:00:05 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:00:05 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:06 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:06 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:06 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:06 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:06 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:06 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:06 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:06 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:06 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:06 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:06 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:06 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:07 fir-io2-s2 kernel: ll_ost_io02_020 D ffff90fec20230c0     0 85054      2 0x00000080
Apr 22 19:00:07 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:00:07 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:00:08 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:00:08 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:08 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:00:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:08 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:00:08 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:08 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:08 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:08 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:08 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:08 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:10 fir-io2-s2 kernel: ll_ost_io03_015 D ffff90fec2024100     0 85055      2 0x00000080
Apr 22 19:00:10 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:00:10 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:11 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:00:11 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:00:11 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:00:11 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:11 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:11 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:11 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:11 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:11 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:11 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:11 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:12 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:12 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:12 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:12 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:12 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:12 fir-io2-s2 kernel: ll_ost_io00_020 D ffff90fec2025140     0 85056      2 0x00000080
Apr 22 19:00:12 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:12 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffc15de03b>] ? ldiskfs_es_insert_extent+0xdb/0x1c0 [ldiskfs]
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:00:13 fir-io2-s2 kernel: [<ffffffffc15ebcd3>] ? ldiskfs_ext_put_gap_in_cache+0x83/0x90 [ldiskfs]
Apr 22 19:00:14 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:00:14 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:00:14 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:00:14 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:00:14 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:00:14 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:00:14 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:14 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:14 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:14 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:14 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:14 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:15 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:15 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:15 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:15 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:15 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:15 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:16 fir-io2-s2 kernel: ll_ost_io00_021 D ffff90fec2026180     0 85057      2 0x00000080
Apr 22 19:00:16 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:16 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:16 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 19:00:16 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:00:16 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:00:16 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:00:16 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:00:16 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:00:16 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:00:16 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:00:16 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:00:16 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 19:00:17 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:00:17 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:00:17 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:00:17 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:00:17 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:17 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:17 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:00:17 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:00:17 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:00:17 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:17 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:17 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:19 fir-io2-s2 kernel: ll_ost_io01_019 D ffff90fc26b58000     0 85058      2 0x00000080
Apr 22 19:00:19 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:19 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:00:19 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:00:19 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:00:19 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:19 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:00:19 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:00:19 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:00:19 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:20 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:20 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:00:20 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:20 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:00:20 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:20 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:20 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:20 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:20 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:20 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:20 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:20 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:21 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:21 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:21 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:21 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:21 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:21 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:21 fir-io2-s2 kernel: ll_ost_io02_021 D ffff90fc26b59040     0 85059      2 0x00000080
Apr 22 19:00:21 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:00:22 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:24 fir-io2-s2 kernel: ll_ost03_074    D ffff90fc26b5a080     0 85060      2 0x00000080
Apr 22 19:00:24 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:24 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:00:24 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:00:24 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:00:24 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:00:24 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:00:24 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:24 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:25 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:25 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:25 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:25 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:25 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:25 fir-io2-s2 kernel: ll_ost03_075    S ffff90fc26b5b0c0     0 85061      2 0x00000080
Apr 22 19:00:25 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:25 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:00:25 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:26 fir-io2-s2 kernel: ll_ost03_076    S ffff90fc26b5c100     0 85062      2 0x00000080
Apr 22 19:00:26 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:27 fir-io2-s2 kernel: ll_ost_io01_020 D ffff90fc26b5d140     0 85064      2 0x00000080
Apr 22 19:00:27 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:27 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:00:27 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:00:27 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:00:27 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:00:27 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:00:27 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:00:27 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:00:27 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:00:27 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:00:28 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:00:28 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:00:28 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:00:28 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:00:28 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:00:28 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:00:28 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:00:28 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:00:28 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:00:28 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:28 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:28 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:29 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:29 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:29 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:29 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:29 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:29 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:29 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:29 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:29 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:29 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:29 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:30 fir-io2-s2 kernel: ll_ost_io00_022 D ffff90fc26b5e180     0 85065      2 0x00000080
Apr 22 19:00:30 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:30 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:31 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:31 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:31 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:31 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:31 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:31 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:31 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:31 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:31 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:31 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:32 fir-io2-s2 kernel: ll_ost03_077    S ffff90fc26bd0000     0 85066      2 0x00000080
Apr 22 19:00:32 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:33 fir-io2-s2 kernel: ll_ost_io02_022 D ffff90fc26bd1040     0 85067      2 0x00000080
Apr 22 19:00:33 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:33 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:00:33 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:00:33 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:00:33 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:00:33 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:00:33 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:00:33 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:00:33 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:00:33 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:00:33 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:00:33 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:34 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:35 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:35 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:35 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:35 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:36 fir-io2-s2 kernel: ll_ost03_078    D ffff90fc26bd2080     0 85080      2 0x00000080
Apr 22 19:00:36 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:36 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:36 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:36 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:00:36 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:00:36 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:00:36 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:00:36 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:00:36 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:36 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:36 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:36 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:36 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:37 fir-io2-s2 kernel: ll_ost03_079    S ffff90fc26bd30c0     0 85083      2 0x00000080
Apr 22 19:00:37 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:38 fir-io2-s2 kernel: ll_ost03_080    D ffff90fc26bd4100     0 85084      2 0x00000080
Apr 22 19:00:38 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:38 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:00:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:00:38 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:00:38 fir-io2-s2 kernel: [<ffffffffa93a0160>] ? __percpu_counter_sum+0x70/0x80
Apr 22 19:00:38 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:00:38 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:00:38 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:00:38 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:38 fir-io2-s2 kernel: Lustre: 84855:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff90f117affc50 x1631526928173696/t0(0) o6->fir-MDT0001-mdtlov_UUID@10.0.10.52@o2ib7:58/0 lens 544/432 e 20 to 0 dl 1555984843 ref 2 fl Interpret:/0/0 rc 0/0
Apr 22 19:00:39 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:39 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:00:39 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:39 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 19:00:39 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 19:00:39 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 19:00:39 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 19:00:39 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 19:00:40 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:40 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:40 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:40 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:40 fir-io2-s2 kernel: ll_ost_io02_023 D ffff90fc26bd5140     0 85085      2 0x00000080
Apr 22 19:00:40 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:41 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:42 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:42 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:42 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:42 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:42 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:42 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:42 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:42 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:42 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:42 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:42 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:43 fir-io2-s2 kernel: ll_ost_io00_023 D ffff90fc26bd6180     0 85086      2 0x00000080
Apr 22 19:00:43 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:43 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:00:43 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:00:43 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:00:43 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:00:43 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:00:43 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:00:43 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:00:43 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:00:44 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 19:00:44 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:00:44 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:00:44 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:00:44 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:00:44 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:44 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:44 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:00:44 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:00:44 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:00:44 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:44 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:46 fir-io2-s2 kernel: ll_ost_io01_021 D ffff91016c5aa080     0 85087      2 0x00000080
Apr 22 19:00:46 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:46 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:00:46 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:00:46 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:00:46 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:00:46 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:00:46 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:46 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:46 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:47 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:47 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:47 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:47 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:47 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:47 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:47 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:47 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:47 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:47 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:47 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:47 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:48 fir-io2-s2 kernel: ll_ost03_081    D ffff91016c5a8000     0 85088      2 0x00000080
Apr 22 19:00:48 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:48 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:00:48 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:00:48 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:00:48 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:00:48 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:00:48 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:48 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:49 fir-io2-s2 kernel: ll_ost_io03_016 D ffff91016c5a9040     0 85089      2 0x00000080
Apr 22 19:00:49 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffa91b9555>] ? mempool_alloc_slab+0x15/0x20
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:00:49 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 19:00:50 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:00:50 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:00:50 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:00:50 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:00:50 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 19:00:50 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:00:50 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 19:00:50 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:00:50 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 19:00:50 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 19:00:50 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:00:50 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:00:51 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 19:00:51 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:00:51 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 19:00:51 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 19:00:51 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 19:00:51 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 19:00:51 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 19:00:51 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 19:00:51 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 19:00:51 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:00:51 fir-io2-s2 kernel: Lustre: 84927:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff90f9d435b450 x1631475204705024/t0(0) o6->fir-MDT0002-mdtlov_UUID@10.0.10.51@o2ib7:71/0 lens 544/432 e 4 to 0 dl 1555984856 ref 2 fl Interpret:/0/0 rc 0/0
Apr 22 19:00:52 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 19:00:52 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:00:52 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:00:52 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:52 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:52 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:52 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:52 fir-io2-s2 kernel: LNet: Service thread pid 84953 was inactive for 537.27s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 19:00:52 fir-io2-s2 kernel: LNet: Skipped 131 previous similar messages
Apr 22 19:00:52 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984852.84953
Apr 22 19:00:52 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:52 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:53 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:53 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:53 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:53 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:53 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:53 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:53 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:53 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:53 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:54 fir-io2-s2 kernel: ll_ost03_082    S ffff91016c5ab0c0     0 85090      2 0x00000080
Apr 22 19:00:54 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:54 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:00:54 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:00:54 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:00:54 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:54 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984854.84745
Apr 22 19:00:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:55 fir-io2-s2 kernel: ll_ost_io02_024 D ffff91016c5ad140     0 85091      2 0x00000080
Apr 22 19:00:55 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:00:55 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:56 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:56 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:00:56 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:00:56 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:00:56 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:56 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:56 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:56 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:56 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:56 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984856.82568
Apr 22 19:00:56 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:56 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:57 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:57 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:57 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:57 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:00:57 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:00:57 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:00:57 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:00:57 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:00:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:00:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:00:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:00:58 fir-io2-s2 kernel: ll_ost_io01_022 D ffff91016c5ae180     0 85092      2 0x00000080
Apr 22 19:00:58 fir-io2-s2 kernel: Call Trace:
Apr 22 19:00:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:00:58 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:00:58 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:00:58 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:00:58 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:00:58 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:00:58 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:00:58 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:00:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:58 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:00:58 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984858.84938
Apr 22 19:00:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:00:58 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:00:59 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:00:59 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:00:59 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:00:59 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:00:59 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:00:59 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:00:59 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:00:59 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:00:59 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:00:59 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:00:59 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:00:59 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:00 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:00 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:00 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:00 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:00 fir-io2-s2 kernel: ll_ost_io01_023 D ffff91016c5ac100     0 85093      2 0x00000080
Apr 22 19:01:00 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:00 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:01:00 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984860.84722
Apr 22 19:01:00 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:01:00 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:01:01 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:01:01 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:01:01 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:01:01 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:01:01 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:01:01 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:01:01 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:01:01 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:01:01 fir-io2-s2 kernel: Lustre: 84927:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff90ff9b1e1850 x1631475204897696/t60129821370(0) o6->fir-MDT0002-mdtlov_UUID@10.0.10.51@o2ib7:81/0 lens 544/432 e 3 to 0 dl 1555984866 ref 2 fl Interpret:/0/0 rc 0/0
Apr 22 19:01:02 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:01:02 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:02 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:02 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:02 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:02 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:02 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:02 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:02 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:02 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:02 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:02 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984862.84795
Apr 22 19:01:03 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:03 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:03 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:03 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:03 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:03 fir-io2-s2 kernel: ll_ost_io02_025 D ffff90f2b79bd140     0 85094      2 0x00000080
Apr 22 19:01:03 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:03 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:01:03 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:01:04 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:01:04 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:01:04 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:01:04 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:04 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:04 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:04 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:04 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:04 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:04 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:04 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:04 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:04 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984864.84870
Apr 22 19:01:05 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:05 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:05 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:05 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:05 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:05 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:05 fir-io2-s2 kernel: ll_ost_io02_026 D ffff90f2b79b9040     0 85095      2 0x00000080
Apr 22 19:01:05 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 19:01:06 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:01:06 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984866.83749
Apr 22 19:01:07 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:07 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:07 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:07 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:07 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:07 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:07 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:08 fir-io2-s2 kernel: ll_ost_io01_024 D ffff90f2b79bb0c0     0 85096      2 0x00000080
Apr 22 19:01:08 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:08 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:01:08 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:01:08 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:01:08 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:01:08 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:01:08 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:08 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:08 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:08 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:08 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:08 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:08 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984868.84814
Apr 22 19:01:09 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:09 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:09 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:09 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:09 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:09 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:09 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:09 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:10 fir-io2-s2 kernel: ll_ost_io01_025 D ffff90f2b79ba080     0 85097      2 0x00000080
Apr 22 19:01:10 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:10 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:01:10 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:01:10 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:01:10 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:01:10 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:01:10 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:01:10 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:01:10 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:01:10 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 19:01:10 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:01:10 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:01:11 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:01:11 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:01:11 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:01:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:01:11 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:01:11 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:01:11 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:01:11 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:11 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:11 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:11 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:11 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:13 fir-io2-s2 kernel: ll_ost_io02_027 D ffff90fed1878000     0 85098      2 0x00000080
Apr 22 19:01:13 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:13 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:01:13 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:01:13 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:01:13 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:01:13 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:01:13 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:01:13 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:01:13 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:01:13 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 19:01:13 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:01:13 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:14 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:15 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:15 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984875.84847
Apr 22 19:01:15 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:15 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:15 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:15 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:16 fir-io2-s2 kernel: ll_ost_io02_028 D ffff90fed1879040     0 85099      2 0x00000080
Apr 22 19:01:16 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:16 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:01:16 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:01:16 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:01:16 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:01:16 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:01:16 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:01:16 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:01:16 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:01:16 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:01:16 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 19:01:16 fir-io2-s2 kernel: [<ffffffffc16615a2>] ? lquota_disk_read+0xf2/0x390 [lquota]
Apr 22 19:01:17 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:01:17 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:01:17 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984877.85156
Apr 22 19:01:17 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:01:17 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:01:17 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:01:17 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:01:17 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:01:17 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:01:17 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:01:17 fir-io2-s2 kernel: Lustre: 85048:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff90e249ec0450 x1630487680490048/t0(0) o4->badfeb6e-c230-04c6-e9c0-74db99ae8754@10.9.101.45@o2ib4:97/0 lens 1688/744 e 13 to 0 dl 1555984882 ref 2 fl Interpret:/0/0 rc 0/0
Apr 22 19:01:17 fir-io2-s2 kernel: Lustre: 85048:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 182 previous similar messages
Apr 22 19:01:18 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:01:18 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:18 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:18 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:18 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:18 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:18 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:18 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:18 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:19 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:19 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:19 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:19 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984879.84965
Apr 22 19:01:19 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:19 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:19 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:20 fir-io2-s2 kernel: ll_ost_io02_029 D ffff90fed187a080     0 85100      2 0x00000080
Apr 22 19:01:20 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:20 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:21 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:21 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:21 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:21 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984881.85458
Apr 22 19:01:21 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:21 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:21 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:21 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:22 fir-io2-s2 kernel: ll_ost_io01_026 D ffff90fed187b0c0     0 85101      2 0x00000080
Apr 22 19:01:22 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:22 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:01:22 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:01:22 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:01:22 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:01:22 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:01:22 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:22 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:22 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:22 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:22 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:23 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:23 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:23 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:23 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:23 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984883.85416
Apr 22 19:01:23 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:23 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:23 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:23 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:23 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:24 fir-io2-s2 kernel: ll_ost_io01_027 D ffff90fed187c100     0 85102      2 0x00000080
Apr 22 19:01:24 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa91581ef>] ? delayacct_end+0x8f/0xb0
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa91c5aa9>] ? mark_page_accessed+0x59/0xa0
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 19:01:24 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:01:25 fir-io2-s2 kernel: [<ffffffffa9765b71>] out_of_line_wait_on_bit_lock+0x81/0xb0
Apr 22 19:01:25 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:01:25 fir-io2-s2 kernel: [<ffffffffa9278f42>] __lock_buffer+0x32/0x40
Apr 22 19:01:25 fir-io2-s2 kernel: [<ffffffffc09e1ea2>] do_get_write_access+0x432/0x4d0 [jbd2]
Apr 22 19:01:25 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 19:01:25 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:01:25 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984885.38497
Apr 22 19:01:25 fir-io2-s2 kernel: [<ffffffffc09ec635>] ? jbd2_journal_add_journal_head+0x65/0x150 [jbd2]
Apr 22 19:01:25 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:01:25 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:01:25 fir-io2-s2 kernel: [<ffffffffc15e1cf8>] ldiskfs_mb_mark_diskspace_used+0xf8/0x510 [ldiskfs]
Apr 22 19:01:25 fir-io2-s2 kernel: [<ffffffffc15e5bb0>] ldiskfs_mb_new_blocks+0x350/0xb20 [ldiskfs]
Apr 22 19:01:25 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 19:01:26 fir-io2-s2 kernel: [<ffffffffc16615a2>] ? lquota_disk_read+0xf2/0x390 [lquota]
Apr 22 19:01:26 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:01:26 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:01:26 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:01:26 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:01:26 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:01:26 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:01:26 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:01:26 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:01:26 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:01:26 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:01:26 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:27 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:27 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:27 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:27 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:27 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:27 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984887.85205
Apr 22 19:01:27 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:27 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:27 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:27 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:27 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:27 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:28 fir-io2-s2 kernel: ll_ost_io02_030 D ffff90fed187d140     0 85103      2 0x00000080
Apr 22 19:01:28 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffa9769535>] rwsem_down_write_failed+0x225/0x3a0
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffa9386c47>] call_rwsem_down_write_failed+0x17/0x30
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffa9766f4d>] down_write+0x2d/0x3d
Apr 22 19:01:28 fir-io2-s2 kernel: [<ffffffffc15ad5b4>] ldiskfs_map_blocks+0x134/0x700 [ldiskfs]
Apr 22 19:01:29 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:01:29 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:01:29 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:01:29 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:01:29 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:29 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984889.38440
Apr 22 19:01:29 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:29 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:29 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:29 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:29 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:29 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:30 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:30 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:30 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:30 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:30 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:30 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:30 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:31 fir-io2-s2 kernel: ll_ost_io01_028 D ffff90fed187e180     0 85104      2 0x00000080
Apr 22 19:01:31 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:31 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:01:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:01:31 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:01:31 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:01:31 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:01:31 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984891.85218
Apr 22 19:01:31 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:01:31 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:01:31 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:01:31 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:32 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:33 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:33 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:33 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:33 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984893.85181
Apr 22 19:01:33 fir-io2-s2 kernel: ll_ost_io02_031 D ffff90feedcd0000     0 85105      2 0x00000080
Apr 22 19:01:33 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:33 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:01:33 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:34 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:35 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:35 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:35 fir-io2-s2 kernel: ll_ost_io01_029 D ffff90feedcd1040     0 85106      2 0x00000080
Apr 22 19:01:35 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:35 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:01:35 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:01:35 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:01:36 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:01:36 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:01:36 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:01:36 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:01:36 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:01:36 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 19:01:36 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:01:36 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:01:36 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:01:36 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:01:36 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:01:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:37 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984897.85210
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:37 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:38 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:38 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:38 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:38 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:38 fir-io2-s2 kernel: ll_ost_io02_032 D ffff90feedcd2080     0 85107      2 0x00000080
Apr 22 19:01:38 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:38 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:40 fir-io2-s2 kernel: ll_ost_io03_017 D ffff90feedcd30c0     0 85108      2 0x00000080
Apr 22 19:01:40 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:40 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:01:40 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:01:40 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:01:40 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:01:40 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:01:40 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:01:40 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:01:40 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:01:40 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:01:41 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:01:41 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:01:41 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:01:41 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:01:41 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:01:41 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:01:41 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:01:41 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:01:41 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:41 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:41 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:41 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:43 fir-io2-s2 kernel: ll_ost_io03_018 D ffff90feedcd4100     0 85109      2 0x00000080
Apr 22 19:01:43 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:43 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 19:01:43 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 19:01:43 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:01:43 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:01:43 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:01:43 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:01:43 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:01:43 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:01:43 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc16763c3>] ? qsd_acquire+0x8e3/0xcb0 [lquota]
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:44 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:45 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:45 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:45 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:45 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:45 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:45 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:45 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:45 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:45 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:45 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:45 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984905.85167
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:46 fir-io2-s2 kernel: ll_ost_io01_030 D ffff90feedcd5140     0 85110      2 0x00000080
Apr 22 19:01:46 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:01:46 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:47 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:47 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:47 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:47 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:47 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:47 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:47 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:47 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:47 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:47 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:47 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:47 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:48 fir-io2-s2 kernel: ll_ost_create02 S ffff90feedcd6180     0 85123      2 0x00000080
Apr 22 19:01:48 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:49 fir-io2-s2 kernel: ll_ost03_083    S ffff90ff33b08000     0 85124      2 0x00000080
Apr 22 19:01:49 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:49 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:01:49 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:01:49 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:01:49 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:49 fir-io2-s2 kernel: Lustre: 85048:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff90e2b549e050 x1631306541440912/t0(0) o4->26ec9cfa-2568-b53b-221d-c03e4e569781@10.9.112.16@o2ib4:129/0 lens 488/0 e 5 to 0 dl 1555984914 ref 2 fl New:/0/ffffffff rc 0/-1
Apr 22 19:01:49 fir-io2-s2 kernel: Lustre: 85048:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1059 previous similar messages
Apr 22 19:01:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:50 fir-io2-s2 kernel: ll_ost_io01_031 D ffff90ff33b09040     0 85125      2 0x00000080
Apr 22 19:01:50 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:50 fir-io2-s2 kernel: [<ffffffffa90dc766>] ? select_task_rq_fair+0x5a6/0x760
Apr 22 19:01:50 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:01:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:50 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:01:50 fir-io2-s2 kernel: [<ffffffffc0bd419d>] ? raid5_compute_blocknr+0xbd/0x290 [raid456]
Apr 22 19:01:50 fir-io2-s2 kernel: [<ffffffffa976778f>] ? __schedule+0x3ff/0x890
Apr 22 19:01:50 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffa9765b71>] out_of_line_wait_on_bit_lock+0x81/0xb0
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffa9278f42>] __lock_buffer+0x32/0x40
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffc09e1ea2>] do_get_write_access+0x432/0x4d0 [jbd2]
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:01:51 fir-io2-s2 kernel: [<ffffffffc15cd6a6>] ldiskfs_quota_write+0xc6/0x1e0 [ldiskfs]
Apr 22 19:01:52 fir-io2-s2 kernel: [<ffffffffa92b2a95>] qtree_write_dquot+0xd5/0x180
Apr 22 19:01:52 fir-io2-s2 kernel: [<ffffffffa92b184b>] v2_write_dquot+0x2b/0x30
Apr 22 19:01:52 fir-io2-s2 kernel: [<ffffffffa92ae36d>] dquot_commit+0xbd/0xd0
Apr 22 19:01:52 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:01:52 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:01:52 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:01:52 fir-io2-s2 kernel: [<ffffffffa92ae6cc>] ? dqput+0x4c/0x1f0
Apr 22 19:01:52 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:01:52 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 19:01:52 fir-io2-s2 kernel: [<ffffffffc16615a2>] ? lquota_disk_read+0xf2/0x390 [lquota]
Apr 22 19:01:52 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:01:52 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:53 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:54 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:54 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:54 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:54 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:54 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:54 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:54 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:54 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:54 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:54 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:55 fir-io2-s2 kernel: ll_ost_io02_033 D ffff90ff33b0a080     0 85126      2 0x00000080
Apr 22 19:01:55 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:55 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:56 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:56 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:56 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:56 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:56 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:56 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:56 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:56 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:56 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:56 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:57 fir-io2-s2 kernel: ll_ost_io01_032 D ffff90ff33b0b0c0     0 85127      2 0x00000080
Apr 22 19:01:57 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:01:57 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:01:57 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:01:57 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:01:57 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:01:57 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:01:57 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:01:57 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:01:57 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:01:58 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:01:59 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:01:59 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:01:59 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:01:59 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:01:59 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:01:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:01:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:01:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:01:59 fir-io2-s2 kernel: ll_ost_io00_024 D ffff90ff33b0c100     0 85128      2 0x00000080
Apr 22 19:01:59 fir-io2-s2 kernel: Call Trace:
Apr 22 19:01:59 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:01:59 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:02:00 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:02:01 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:02:01 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:01 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:01 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:01 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:01 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:01 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:01 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:01 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:01 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:01 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:01 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:02 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:02 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:02 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:02 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:02 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:02 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:02 fir-io2-s2 kernel: ll_ost_io02_034 D ffff90ff33b0d140     0 85129      2 0x00000080
Apr 22 19:02:02 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:02 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:02:02 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:02:02 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:03 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:04 fir-io2-s2 kernel: ll_ost_io00_025 D ffff90ff33b0e180     0 85130      2 0x00000080
Apr 22 19:02:04 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:02:04 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:02:05 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:02:05 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:02:05 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:02:05 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:02:05 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:02:05 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:02:05 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:02:05 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:02:05 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:02:05 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:02:05 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:02:05 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:02:06 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:02:06 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:02:06 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:06 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:06 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:06 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:06 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:06 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:06 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:06 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:06 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:06 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:07 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:07 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:07 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:07 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:07 fir-io2-s2 kernel: ll_ost_io01_033 D ffff9100ea700000     0 85131      2 0x00000080
Apr 22 19:02:07 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:07 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:02:07 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:08 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:02:08 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:08 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:02:08 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:02:08 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:02:08 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984928.84733
Apr 22 19:02:08 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:08 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:02:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:08 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:02:08 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:09 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:10 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:10 fir-io2-s2 kernel: ll_ost_io00_026 D ffff9100ea701040     0 85132      2 0x00000080
Apr 22 19:02:10 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984930.82415
Apr 22 19:02:10 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:10 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:02:10 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:10 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:02:10 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:11 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:12 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:12 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:12 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:12 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:12 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:12 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984932.84932
Apr 22 19:02:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:12 fir-io2-s2 kernel: [<ffffffffa90d0884>] ? finish_task_switch+0x54/0x1c0
Apr 22 19:02:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:13 fir-io2-s2 kernel: ll_ost_io02_035 D ffff9100ea702080     0 85133      2 0x00000080
Apr 22 19:02:13 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:13 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:02:13 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:13 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:02:13 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:02:13 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:02:13 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:02:13 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:13 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:13 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:02:13 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:14 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:15 fir-io2-s2 kernel: ll_ost_io03_019 D ffff9100ea7030c0     0 85134      2 0x00000080
Apr 22 19:02:15 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:15 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:02:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:15 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:02:15 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:02:16 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:02:16 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:02:16 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:16 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:02:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:16 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:02:16 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984936.84915
Apr 22 19:02:16 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:16 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:16 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:16 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:17 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:17 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:17 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:17 fir-io2-s2 kernel: Lustre: fir-OST0011: Connection restored to ca202028-44b5-b8ef-64ec-668c106c6135 (at 10.8.21.23@o2ib6)
Apr 22 19:02:17 fir-io2-s2 kernel: Lustre: Skipped 324 previous similar messages
Apr 22 19:02:17 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:17 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:17 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:18 fir-io2-s2 kernel: ll_ost_io01_034 D ffff9100ea704100     0 85135      2 0x00000080
Apr 22 19:02:18 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:18 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:02:18 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:02:18 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984938.84727
Apr 22 19:02:18 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:18 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:02:18 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:02:18 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:19 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:19 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:19 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:19 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:19 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:19 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:19 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:19 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:19 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:19 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:19 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:19 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:20 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:20 fir-io2-s2 kernel: ll_ost_io00_027 D ffff9100ea705140     0 85136      2 0x00000080
Apr 22 19:02:20 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:20 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:02:20 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:02:20 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984940.84754
Apr 22 19:02:20 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:02:20 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:02:20 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:02:20 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:02:21 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:02:21 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:02:21 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:02:21 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:02:21 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:02:21 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:02:21 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:21 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:21 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:02:21 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:02:21 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:02:21 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:22 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:22 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:22 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:22 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:22 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:22 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:22 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:22 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:22 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:22 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:22 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984942.84775
Apr 22 19:02:22 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:22 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:23 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:23 fir-io2-s2 kernel: ll_ost_io01_035 D ffff9100ea706180     0 85137      2 0x00000080
Apr 22 19:02:23 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:23 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:02:23 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:02:23 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:02:23 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:02:23 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:02:24 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:02:24 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:02:24 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:02:24 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:02:24 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:02:24 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:02:24 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:02:24 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:02:24 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:02:24 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:02:24 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984944.82867
Apr 22 19:02:24 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:02:24 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:02:25 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:02:25 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:25 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:25 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:25 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:25 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:25 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:25 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:25 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:25 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:25 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:25 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:26 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:26 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:26 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:26 fir-io2-s2 kernel: ll_ost_io00_028 D ffff9100ea438000     0 85138      2 0x00000080
Apr 22 19:02:26 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:26 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:02:26 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984946.84821
Apr 22 19:02:26 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:02:26 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:27 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:02:27 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:02:27 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:27 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:27 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:27 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:27 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:27 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:27 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:27 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:27 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:27 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984947.83755
Apr 22 19:02:28 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:28 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:28 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:28 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:28 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:28 fir-io2-s2 kernel: ll_ost_io01_036 D ffff9100ea439040     0 85139      2 0x00000080
Apr 22 19:02:28 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:28 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984948.84741
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:02:29 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:30 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:30 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:30 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:30 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:30 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:30 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:30 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:30 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:30 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:30 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:30 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:30 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:31 fir-io2-s2 kernel: ll_ost_io02_036 D ffff9100ea43a080     0 85140      2 0x00000080
Apr 22 19:02:31 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:31 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:02:31 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:02:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:31 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:02:31 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:32 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:32 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984952.83794
Apr 22 19:02:33 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:33 fir-io2-s2 kernel: ll_ost_io02_037 D ffff9100ea43b0c0     0 85141      2 0x00000080
Apr 22 19:02:33 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:33 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:02:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:33 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:02:33 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:02:33 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:02:34 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:02:34 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:34 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:02:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:34 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:02:34 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:34 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:34 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:34 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:34 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:35 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:35 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984954.84908
Apr 22 19:02:35 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:35 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:35 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:35 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:36 fir-io2-s2 kernel: ll_ost_io01_037 D ffff9100ea43c100     0 85142      2 0x00000080
Apr 22 19:02:36 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:36 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:36 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 19:02:36 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:02:36 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:02:36 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:02:36 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:02:36 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:02:36 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:02:36 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:02:36 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:02:36 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:02:37 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:02:37 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:02:37 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984957.84891
Apr 22 19:02:37 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:02:37 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:37 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:37 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:02:37 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:02:37 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:02:37 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:37 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:37 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:37 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:39 fir-io2-s2 kernel: ll_ost_io01_038 D ffff9100ea43d140     0 85143      2 0x00000080
Apr 22 19:02:39 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:39 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:02:39 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:02:39 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:02:39 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:02:39 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:02:39 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:02:39 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:02:39 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:02:39 fir-io2-s2 kernel: [<ffffffffc15de03b>] ? ldiskfs_es_insert_extent+0xdb/0x1c0 [ldiskfs]
Apr 22 19:02:39 fir-io2-s2 kernel: [<ffffffffc15dd800>] ? __es_tree_search+0x60/0x80 [ldiskfs]
Apr 22 19:02:40 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:02:40 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:02:40 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:02:40 fir-io2-s2 kernel: [<ffffffffc15ebcd3>] ? ldiskfs_ext_put_gap_in_cache+0x83/0x90 [ldiskfs]
Apr 22 19:02:40 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:02:40 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:02:40 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:02:40 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:02:40 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:02:40 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:02:40 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:40 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:41 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:41 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:41 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:41 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:41 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:41 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:41 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:41 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:41 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:41 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:41 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:41 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:42 fir-io2-s2 kernel: ll_ost_io00_029 D ffff9100ea43e180     0 85144      2 0x00000080
Apr 22 19:02:42 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:42 fir-io2-s2 kernel: [<ffffffffc1663e36>] ? lqe_locate+0x46/0x840 [lquota]
Apr 22 19:02:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:42 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:02:42 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:42 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:02:42 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:02:42 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 19:02:43 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:44 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:44 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:44 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:44 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:02:44 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:44 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:44 fir-io2-s2 kernel: ll_ost_io02_038 D ffff90fef7378000     0 85145      2 0x00000080
Apr 22 19:02:44 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:45 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:02:45 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:45 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:02:45 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:02:45 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:02:45 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:02:45 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:45 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:45 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:02:45 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:45 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:02:45 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:46 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:47 fir-io2-s2 kernel: ll_ost_io01_039 D ffff90fef7379040     0 85146      2 0x00000080
Apr 22 19:02:47 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:47 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:02:47 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:02:47 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:47 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:02:47 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:02:47 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:48 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:49 fir-io2-s2 kernel: ldlm_cn00_005   S ffff90e2b99d6180     0 85147      2 0x00000080
Apr 22 19:02:49 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:49 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:02:49 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:02:49 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:02:49 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:50 fir-io2-s2 kernel: ll_ost_io02_039 D ffff9100d7978000     0 85148      2 0x00000080
Apr 22 19:02:50 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:50 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:02:50 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:50 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:02:50 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:02:50 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:02:50 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:02:50 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:50 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:52 fir-io2-s2 kernel: ll_ost_io00_030 D ffff9100d7979040     0 85149      2 0x00000080
Apr 22 19:02:52 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:52 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:02:52 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:52 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:02:52 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:02:52 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:02:52 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:02:52 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:52 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:53 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:02:53 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:53 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:02:53 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:02:53 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:53 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:53 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:53 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:53 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:53 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:53 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:53 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:54 fir-io2-s2 kernel: Lustre: 85048:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff90e2b59cd050 x1631541808832768/t0(0) o3->dbdaea0c-6cd1-174f-909a-de8e042f528b@10.9.108.23@o2ib4:193/0 lens 488/0 e 4 to 0 dl 1555984978 ref 2 fl New:/0/ffffffff rc 0/-1
Apr 22 19:02:54 fir-io2-s2 kernel: Lustre: 85048:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 1163 previous similar messages
Apr 22 19:02:54 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:54 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:54 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:54 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:54 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:54 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:55 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:55 fir-io2-s2 kernel: ll_ost_io02_040 D ffff9100d797a080     0 85150      2 0x00000080
Apr 22 19:02:55 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:02:55 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:02:55 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:02:55 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:02:55 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:02:55 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:02:55 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:56 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:02:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:02:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:02:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:02:57 fir-io2-s2 kernel: ll_ost_io01_040 D ffff9100d797b0c0     0 85151      2 0x00000080
Apr 22 19:02:57 fir-io2-s2 kernel: Call Trace:
Apr 22 19:02:57 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:02:57 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:02:57 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:02:57 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:02:57 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:02:57 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:02:57 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:02:58 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:02:59 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:02:59 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:02:59 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:02:59 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:02:59 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:02:59 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:02:59 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:02:59 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984979.85376
Apr 22 19:02:59 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:02:59 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:02:59 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:02:59 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:00 fir-io2-s2 kernel: ll_ost_io00_031 D ffff9100d797c100     0 85152      2 0x00000080
Apr 22 19:03:00 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:00 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:03:00 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:00 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984980.84761
Apr 22 19:03:00 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:03:00 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:03:00 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:03:00 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:03:01 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:01 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:01 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:03:01 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:01 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:03:01 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:03:01 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:01 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:01 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555984981.82393
Apr 22 19:03:01 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:01 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:01 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:01 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:02 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:02 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:02 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:02 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:02 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:02 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:02 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:02 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:02 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:02 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:02 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:03 fir-io2-s2 kernel: ll_ost_io02_041 D ffff9100d797d140     0 85153      2 0x00000080
Apr 22 19:03:03 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:03 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:03:03 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:03:03 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:03:03 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:03:03 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:03:03 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:03:03 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:03:03 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:03:03 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 19:03:03 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:03:03 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:03:04 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:03:04 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:03:04 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:04 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:04 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:03:04 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:03:04 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:03:04 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:04 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:04 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:04 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:04 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:06 fir-io2-s2 kernel: ll_ost_io00_032 D ffff9100d797e180     0 85154      2 0x00000080
Apr 22 19:03:06 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:06 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:03:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:06 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:03:06 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:03:06 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:03:06 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:03:06 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:06 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:06 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:03:06 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:07 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:03:07 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:03:07 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:07 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:07 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:07 fir-io2-s2 kernel: [<ffffffffa976778f>] ? __schedule+0x3ff/0x890
Apr 22 19:03:07 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:07 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:07 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:07 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:07 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:07 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:08 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:08 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:08 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:08 fir-io2-s2 kernel: ll_ost_io02_042 D ffff9100e4f10000     0 85155      2 0x00000080
Apr 22 19:03:08 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:08 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:03:08 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:08 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:03:08 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:09 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:11 fir-io2-s2 kernel: ll_ost_io01_041 D ffff9100e4f11040     0 85156      2 0x00000080
Apr 22 19:03:11 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:11 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:03:11 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:03:11 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:11 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:03:11 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:03:11 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:11 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:11 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:11 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:11 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:12 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:12 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:12 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:12 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:12 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:12 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:12 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:12 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:13 fir-io2-s2 kernel: ll_ost_io02_043 D ffff9100e4f12080     0 85157      2 0x00000080
Apr 22 19:03:13 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:03:13 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:14 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:03:14 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:03:14 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:14 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:14 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:14 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:14 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:14 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:14 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:14 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:14 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:15 fir-io2-s2 kernel: ll_ost_io00_033 D ffff9100e4f130c0     0 85158      2 0x00000080
Apr 22 19:03:15 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:15 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:03:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:15 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:03:15 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:03:16 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:03:16 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:03:16 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:16 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:03:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:16 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:03:16 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:03:16 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:16 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:16 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:16 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:18 fir-io2-s2 kernel: ll_ost_io00_034 D ffff9100e4f14100     0 85159      2 0x00000080
Apr 22 19:03:18 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:18 fir-io2-s2 kernel: [<ffffffffa921a849>] ? ___slab_alloc+0x209/0x4f0
Apr 22 19:03:18 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:03:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:18 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:03:18 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 19:03:18 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:03:18 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:03:18 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:03:18 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:03:18 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 19:03:18 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:03:19 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 19:03:19 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:03:19 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 19:03:19 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 19:03:19 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:03:19 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:03:19 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 19:03:19 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:03:19 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 19:03:19 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 19:03:19 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 19:03:19 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:20 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:22 fir-io2-s2 kernel: ll_ost_io01_042 D ffff9100e4f15140     0 85160      2 0x00000080
Apr 22 19:03:22 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:22 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985002.84811
Apr 22 19:03:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:22 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:03:22 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:03:22 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:22 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:03:22 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:03:22 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:22 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:22 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:22 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:22 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:23 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 19:03:23 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:23 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:23 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:23 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:23 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:03:23 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:24 fir-io2-s2 kernel: ll_ost_io01_043 D ffff9100e4f16180     0 85161      2 0x00000080
Apr 22 19:03:24 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:03:24 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985004.84771
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:03:24 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:03:25 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:25 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:25 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:03:25 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:03:25 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:03:25 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:25 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:25 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:25 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:25 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:25 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:25 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:26 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:26 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:26 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:26 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985006.83784
Apr 22 19:03:26 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:26 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:26 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:26 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:27 fir-io2-s2 kernel: ll_ost_io01_044 D ffff9100e4ff0000     0 85162      2 0x00000080
Apr 22 19:03:27 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:27 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:28 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:28 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:28 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:28 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:28 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:28 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:28 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:28 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:29 fir-io2-s2 kernel: ll_ost_io01_045 D ffff9100e4ff1040     0 85163      2 0x00000080
Apr 22 19:03:29 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:29 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:03:29 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:03:29 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:03:29 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:03:29 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:03:29 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:03:29 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:03:29 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:03:29 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:03:29 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:03:29 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:03:30 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:03:30 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:30 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:30 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:03:30 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985010.85044
Apr 22 19:03:30 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:03:30 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:03:30 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:30 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:30 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:30 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:30 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:30 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:31 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:31 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:31 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:31 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:31 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:31 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:31 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:32 fir-io2-s2 kernel: ll_ost03_084    S ffff9100e4ff2080     0 85164      2 0x00000080
Apr 22 19:03:32 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:32 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:03:32 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:03:32 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985012.84866
Apr 22 19:03:32 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:03:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:32 fir-io2-s2 kernel: ll_ost03_085    S ffff9100e4ff30c0     0 85165      2 0x00000080
Apr 22 19:03:32 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:33 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:03:33 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:03:33 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:03:33 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:33 fir-io2-s2 kernel: ll_ost03_086    D ffff9100e4ff4100     0 85166      2 0x00000080
Apr 22 19:03:33 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:33 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:03:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:34 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:03:34 fir-io2-s2 kernel: [<ffffffffa93a0160>] ? __percpu_counter_sum+0x70/0x80
Apr 22 19:03:34 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:03:34 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:03:34 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:03:34 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:34 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:03:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:34 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 19:03:34 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 19:03:34 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:36 fir-io2-s2 kernel: ll_ost03_087    D ffff9100e4ff5140     0 85167      2 0x00000080
Apr 22 19:03:36 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:36 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:36 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:03:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:36 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:03:36 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:03:36 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:03:36 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985016.84806
Apr 22 19:03:36 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:03:36 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:36 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffc1298863>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:38 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:38 fir-io2-s2 kernel: Lustre: fir-OST0011: Client 510ec457-a953-0a53-cbfa-dcd227c2ed46 (at 10.9.105.57@o2ib4) reconnecting
Apr 22 19:03:38 fir-io2-s2 kernel: Lustre: Skipped 1533 previous similar messages
Apr 22 19:03:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:38 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985018.84823
Apr 22 19:03:38 fir-io2-s2 kernel: ll_ost_io02_044 D ffff9100e4ff6180     0 85168      2 0x00000080
Apr 22 19:03:38 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:38 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:03:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:38 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:03:38 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 19:03:38 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:03:39 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 19:03:40 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 19:03:40 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 19:03:40 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 19:03:40 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 19:03:40 fir-io2-s2 kernel: [<ffffffffa9269218>] ? generic_setxattr+0x68/0x80
Apr 22 19:03:40 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 19:03:40 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 19:03:40 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985020.84883
Apr 22 19:03:40 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:03:40 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:03:40 fir-io2-s2 kernel: [<ffffffffc180892b>] ofd_object_punch+0x4cb/0xd30 [ofd]
Apr 22 19:03:40 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:03:41 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:41 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:41 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:41 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:41 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:41 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:41 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:41 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:42 fir-io2-s2 kernel: ll_ost_io01_046 D ffff9100e0f20000     0 85169      2 0x00000080
Apr 22 19:03:42 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:42 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:03:42 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:42 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:03:42 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:03:42 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:03:42 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:03:42 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:42 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985022.84842
Apr 22 19:03:42 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:42 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:03:42 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:42 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:03:43 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:43 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:43 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:43 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:43 fir-io2-s2 kernel: Lustre: fir-OST0017: Export ffff911264395c00 already connecting from 10.8.7.15@o2ib6
Apr 22 19:03:43 fir-io2-s2 kernel: Lustre: Skipped 30 previous similar messages
Apr 22 19:03:43 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:43 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:43 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:43 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:43 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:43 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:44 fir-io2-s2 kernel: ll_ost_io01_047 D ffff9100e0f21040     0 85170      2 0x00000080
Apr 22 19:03:44 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffa927882d>] ? __brelse+0x3d/0x50
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffa9278b1a>] ? bh_lru_install+0x18a/0x1e0
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:44 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffc15b0dc0>] ldiskfs_reserve_inode_write+0x70/0xa0 [ldiskfs]
Apr 22 19:03:45 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ? ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 19:03:46 fir-io2-s2 kernel: [<ffffffffc15b0e43>] ldiskfs_mark_inode_dirty+0x53/0x210 [ldiskfs]
Apr 22 19:03:46 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 19:03:46 fir-io2-s2 kernel: [<ffffffffc16c68f1>] osd_attr_set+0x3f1/0xb90 [osd_ldiskfs]
Apr 22 19:03:46 fir-io2-s2 kernel: [<ffffffffc16e73db>] ? osd_punch+0xcb/0x6b0 [osd_ldiskfs]
Apr 22 19:03:46 fir-io2-s2 kernel: [<ffffffffc1808d68>] ofd_object_punch+0x908/0xd30 [ofd]
Apr 22 19:03:46 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:03:46 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:46 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:46 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985026.84859
Apr 22 19:03:46 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:46 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:47 fir-io2-s2 kernel: ll_ost_io03_020 D ffff9100e0f22080     0 85171      2 0x00000080
Apr 22 19:03:47 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:03:47 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:03:48 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:03:48 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:03:48 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:03:48 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:03:48 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:03:48 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:03:48 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:03:48 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:48 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:48 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:03:48 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985028.85001
Apr 22 19:03:48 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:03:48 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:03:49 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:49 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:49 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:49 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:49 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:49 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:49 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:49 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:49 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:49 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:49 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:49 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:50 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:50 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:50 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:50 fir-io2-s2 kernel: ll_ost_io00_035 D ffff9100e0f230c0     0 85172      2 0x00000080
Apr 22 19:03:50 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:50 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:03:50 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:50 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:03:50 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:03:51 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:03:51 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:03:51 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:51 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:51 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:03:51 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:51 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:03:51 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:51 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:51 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:51 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:51 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:53 fir-io2-s2 kernel: ll_ost_io02_045 D ffff9100e0f24100     0 85173      2 0x00000080
Apr 22 19:03:53 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:53 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:03:53 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:03:53 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:03:53 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:03:53 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:03:53 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:03:53 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:03:53 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:03:53 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:03:53 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:03:53 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:54 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:55 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:55 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:55 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:55 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:55 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:55 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:55 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:56 fir-io2-s2 kernel: ll_ost_io00_036 D ffff9100e0f25140     0 85174      2 0x00000080
Apr 22 19:03:56 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:03:56 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:03:57 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:57 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:57 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:57 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:03:57 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:03:57 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:03:57 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:03:57 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:03:57 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:03:57 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:03:57 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:03:57 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:03:58 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:03:58 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:03:58 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:03:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:03:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:03:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:03:58 fir-io2-s2 kernel: ll_ost_io02_046 D ffff9100e0f26180     0 85175      2 0x00000080
Apr 22 19:03:58 fir-io2-s2 kernel: Call Trace:
Apr 22 19:03:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:03:58 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:03:58 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:03:58 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:03:58 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:03:59 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:03:59 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:03:59 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:03:59 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:59 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:03:59 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:03:59 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:03:59 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:03:59 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:03:59 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:03:59 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:03:59 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:01 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985041.84755
Apr 22 19:04:01 fir-io2-s2 kernel: ll_ost_io01_048 D ffff90fefcbf8000     0 85176      2 0x00000080
Apr 22 19:04:01 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:01 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:04:01 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:04:01 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:04:01 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:04:01 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:04:01 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:04:01 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:04:01 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:04:01 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:04:01 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:04:01 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:04:01 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:04:02 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:04:02 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:02 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:04:02 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:04:02 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:04:02 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:02 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:02 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:02 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:02 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:02 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:02 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffa90d0884>] ? finish_task_switch+0x54/0x1c0
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:04 fir-io2-s2 kernel: ll_ost_io00_037 D ffff90fefcbf9040     0 85177      2 0x00000080
Apr 22 19:04:04 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa90c2cc5>] ? wake_up_bit+0x25/0x30
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa9765b71>] out_of_line_wait_on_bit_lock+0x81/0xb0
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:04:04 fir-io2-s2 kernel: [<ffffffffa9278f42>] __lock_buffer+0x32/0x40
Apr 22 19:04:05 fir-io2-s2 kernel: [<ffffffffc09e1ea2>] do_get_write_access+0x432/0x4d0 [jbd2]
Apr 22 19:04:05 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 19:04:05 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:04:05 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:04:05 fir-io2-s2 kernel: [<ffffffffc15e1c7d>] ldiskfs_mb_mark_diskspace_used+0x7d/0x510 [ldiskfs]
Apr 22 19:04:05 fir-io2-s2 kernel: [<ffffffffc15e5bb0>] ldiskfs_mb_new_blocks+0x350/0xb20 [ldiskfs]
Apr 22 19:04:05 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:04:05 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:04:05 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:04:05 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:04:05 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:04:05 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:04:06 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:04:06 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:04:06 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:04:06 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:04:06 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:06 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:06 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:06 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:06 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:06 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:06 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:06 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:07 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:07 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:07 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:07 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:07 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:07 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:07 fir-io2-s2 kernel: ll_ost_io01_049 D ffff90fefcbfa080     0 85178      2 0x00000080
Apr 22 19:04:07 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:08 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:04:08 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:04:08 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:04:08 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:04:08 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:04:08 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:04:08 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:04:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:08 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:04:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:08 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:04:08 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:09 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:10 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:10 fir-io2-s2 kernel: ll_ost_io00_038 D ffff90fefcbfb0c0     0 85179      2 0x00000080
Apr 22 19:04:10 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:10 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:04:10 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:04:10 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:04:10 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:04:10 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:04:10 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:04:11 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:04:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:11 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:04:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:11 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:04:11 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:04:11 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:11 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:11 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:11 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:11 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:11 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:13 fir-io2-s2 kernel: ll_ost_io02_047 D ffff90fefcbfc100     0 85180      2 0x00000080
Apr 22 19:04:13 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:13 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:04:13 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:04:13 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:04:13 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:04:13 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:04:13 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:04:13 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:04:13 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:13 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:04:13 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:14 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:04:14 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:04:14 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:14 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:14 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:14 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:14 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:14 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:14 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:14 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:14 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:15 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:15 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:15 fir-io2-s2 kernel: ll_ost_io00_039 D ffff90fefcbfd140     0 85181      2 0x00000080
Apr 22 19:04:15 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:15 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:04:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:16 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:17 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:17 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:17 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:17 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:17 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:17 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:17 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985057.84934
Apr 22 19:04:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:18 fir-io2-s2 kernel: ll_ost03_088    D ffff90fefcbfe180     0 85182      2 0x00000080
Apr 22 19:04:18 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:18 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:04:18 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:04:18 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:04:18 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:04:18 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:04:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:18 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:19 fir-io2-s2 kernel: ll_ost03_089    S ffff90febcdf8000     0 85183      2 0x00000080
Apr 22 19:04:19 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:04:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:20 fir-io2-s2 kernel: ll_ost03_090    D ffff90febcdf9040     0 85184      2 0x00000080
Apr 22 19:04:20 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa93824d9>] ? pointer.isra.19+0x1c9/0x4d0
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa9382a14>] ? vsnprintf+0x234/0x6a0
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:04:20 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:04:21 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:04:21 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 19:04:21 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:04:21 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 19:04:21 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:04:21 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 19:04:21 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 19:04:21 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:04:21 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:04:21 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 19:04:21 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 19:04:21 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffc1298909>] tgt_client_data_update+0x3a9/0x5e0 [ptlrpc]
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffc1298f5b>] tgt_client_new+0x41b/0x610 [ptlrpc]
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffc18007d3>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffc11ff62e>] target_handle_connect+0x109e/0x2950 [ptlrpc]
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffc12a6b2a>] tgt_request_handle+0x50a/0x1580 [ptlrpc]
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:22 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:23 fir-io2-s2 kernel: ll_ost_io01_050 D ffff90febcdfa080     0 85185      2 0x00000080
Apr 22 19:04:23 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:23 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:04:23 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:04:23 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:04:23 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:04:23 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:04:23 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:04:24 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:04:24 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:24 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:04:24 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:24 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:04:24 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 19:04:24 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:04:24 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:24 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:24 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:24 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:24 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:25 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:25 fir-io2-s2 kernel: ll_ost_io02_048 D ffff90febcdfb0c0     0 85186      2 0x00000080
Apr 22 19:04:25 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:25 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:04:25 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:04:25 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:04:25 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:25 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:04:25 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:26 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:27 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:27 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:27 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:27 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:27 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:27 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:27 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:28 fir-io2-s2 kernel: ll_ost03_091    S ffff90febcdfc100     0 85187      2 0x00000080
Apr 22 19:04:28 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:28 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:04:28 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:04:28 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:04:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:28 fir-io2-s2 kernel: ll_ost03_092    S ffff90febcdfd140     0 85188      2 0x00000080
Apr 22 19:04:28 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:28 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:04:28 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:04:29 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:04:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:29 fir-io2-s2 kernel: ll_ost_io03_021 D ffff90febcdfe180     0 85189      2 0x00000080
Apr 22 19:04:29 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:29 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:04:29 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:04:29 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:04:29 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:30 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:31 fir-io2-s2 kernel: ll_ost03_093    S ffff90feafb28000     0 85190      2 0x00000080
Apr 22 19:04:31 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:31 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:04:31 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:04:31 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:04:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:32 fir-io2-s2 kernel: ll_ost_io03_022 D ffff90feafb29040     0 85191      2 0x00000080
Apr 22 19:04:32 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:04:32 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:33 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:04:33 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:33 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:04:33 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:04:33 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:33 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:33 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:33 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:33 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:33 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:33 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:33 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:34 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:34 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:34 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:34 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:34 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:34 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:34 fir-io2-s2 kernel: ll_ost03_094    D ffff90feafb2a080     0 85192      2 0x00000080
Apr 22 19:04:34 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:34 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:04:35 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:04:35 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:04:35 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:04:35 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:04:35 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:35 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:35 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:35 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:36 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:36 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:36 fir-io2-s2 kernel: ll_ost_io01_051 D ffff90feafb2b0c0     0 85193      2 0x00000080
Apr 22 19:04:36 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:36 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 19:04:36 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 19:04:36 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:04:36 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:04:36 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:04:36 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:04:36 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:04:36 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:04:36 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:04:37 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:04:37 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:04:37 fir-io2-s2 kernel: [<ffffffffc122c3e0>] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc]
Apr 22 19:04:37 fir-io2-s2 kernel: [<ffffffffc1668347>] ? qsd_send_dqacq+0x307/0x340 [lquota]
Apr 22 19:04:37 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:04:37 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:04:37 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:04:37 fir-io2-s2 kernel: [<ffffffffc16763c3>] ? qsd_acquire+0x8e3/0xcb0 [lquota]
Apr 22 19:04:37 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:04:37 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:04:37 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:04:37 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985077.83731
Apr 22 19:04:38 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:04:38 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:04:38 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:04:38 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:38 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:38 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:38 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:38 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:38 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:38 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:38 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:38 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:39 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:39 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:39 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:39 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:39 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:39 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:39 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:39 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:39 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:39 fir-io2-s2 kernel: ll_ost_io02_049 D ffff90feafb2c100     0 85194      2 0x00000080
Apr 22 19:04:39 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:39 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:04:39 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:04:39 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985079.83792
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:04:40 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:41 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:04:41 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:04:41 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:04:41 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:41 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:41 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:41 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:41 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:41 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:41 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:41 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:41 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:42 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:42 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:42 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:42 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:42 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:42 fir-io2-s2 kernel: ll_ost03_095    D ffff90feafb2d140     0 85195      2 0x00000080
Apr 22 19:04:42 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:42 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:04:42 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:44 fir-io2-s2 kernel: ll_ost03_096    S ffff90feafb2e180     0 85196      2 0x00000080
Apr 22 19:04:44 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:44 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:04:44 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:04:44 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:04:44 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:44 fir-io2-s2 kernel: ll_ost_io00_040 D ffff90feb8338000     0 85197      2 0x00000080
Apr 22 19:04:44 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:44 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:04:44 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:04:45 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:04:45 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:04:45 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:04:45 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:04:45 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:04:45 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:45 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:04:45 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:45 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:04:45 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:04:45 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:45 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:46 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:46 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:46 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:46 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 19:04:46 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:46 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:46 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:46 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:46 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:46 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:46 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:46 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:47 fir-io2-s2 kernel: ll_ost_io03_023 D ffff90feb8339040     0 85198      2 0x00000080
Apr 22 19:04:47 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:04:47 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:48 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:04:48 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:48 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:04:48 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:04:48 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:48 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:48 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:48 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:48 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:48 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:48 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:48 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:49 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:49 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:49 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:49 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:49 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:49 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:49 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:49 fir-io2-s2 kernel: ll_ost03_097    D ffff90feb833a080     0 85199      2 0x00000080
Apr 22 19:04:49 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:49 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:04:50 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:04:50 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:04:50 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:04:50 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:04:50 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:50 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:50 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:50 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:50 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:50 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:50 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:51 fir-io2-s2 kernel: ll_ost_io00_041 D ffff90feb833b0c0     0 85200      2 0x00000080
Apr 22 19:04:51 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:04:51 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:04:51 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:04:51 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:04:51 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:04:51 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:04:51 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:04:51 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:04:51 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:52 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:04:52 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:04:52 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:04:52 fir-io2-s2 kernel: LNet: Service thread pid 84828 was inactive for 576.76s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Apr 22 19:04:52 fir-io2-s2 kernel: LNet: Skipped 4 previous similar messages
Apr 22 19:04:52 fir-io2-s2 kernel: Pid: 84828, comm: ll_ost01_087 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 19:04:52 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:52 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:04:52 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:04:52 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:52 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:52 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 19:04:52 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985092.84828
Apr 22 19:04:53 fir-io2-s2 kernel: Pid: 84803, comm: ll_ost01_076 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 19:04:53 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:53 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:04:53 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:04:53 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:53 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:53 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 19:04:53 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 19:04:53 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:04:54 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:54 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:54 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:54 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:54 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:54 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:54 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:54 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:55 fir-io2-s2 kernel: ll_ost_io01_052 D ffff90feb833c100     0 85201      2 0x00000080
Apr 22 19:04:55 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:55 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:04:55 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:04:55 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:04:55 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:04:55 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:04:55 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:04:55 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:04:55 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:04:55 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:04:55 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 19:04:55 fir-io2-s2 kernel: [<ffffffffc16615a2>] ? lquota_disk_read+0xf2/0x390 [lquota]
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:04:56 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:04:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:04:58 fir-io2-s2 kernel: ll_ost03_098    D ffff90feb833d140     0 85202      2 0x00000080
Apr 22 19:04:58 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:58 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:04:58 fir-io2-s2 kernel: Pid: 84875, comm: ll_ost00_063 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 19:04:58 fir-io2-s2 kernel: Call Trace:
Apr 22 19:04:58 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:04:58 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:04:58 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:58 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:04:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:04:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:04:58 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 19:04:58 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985098.84875
Apr 22 19:04:59 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:04:59 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:04:59 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:04:59 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:04:59 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:04:59 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:04:59 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:04:59 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:04:59 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:04:59 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:00 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:00 fir-io2-s2 kernel: ll_ost_io01_053 D ffff90feb833e180     0 85203      2 0x00000080
Apr 22 19:05:00 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:00 fir-io2-s2 kernel: [<ffffffffa959a485>] bitmap_startwrite+0x1f5/0x210
Apr 22 19:05:00 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:00 fir-io2-s2 kernel: [<ffffffffc0bcf2e1>] add_stripe_bio+0x421/0x7d0 [raid456]
Apr 22 19:05:00 fir-io2-s2 kernel: [<ffffffffc0bd4c34>] raid5_make_request+0x1e4/0xc70 [raid456]
Apr 22 19:05:00 fir-io2-s2 kernel: [<ffffffffc15ecd00>] ? __ldiskfs_ext_dirty+0x40/0xa0 [ldiskfs]
Apr 22 19:05:00 fir-io2-s2 kernel: [<ffffffffc15ecd33>] ? __ldiskfs_ext_dirty+0x73/0xa0 [ldiskfs]
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffa958d330>] md_handle_request+0xd0/0x150
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffa934311f>] ? generic_make_request_checks+0x27f/0x390
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffa958d499>] md_make_request+0x79/0x190
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffa9344ef7>] generic_make_request+0x147/0x380
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffa9587a86>] ? md_mergeable_bvec+0x46/0x50
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffa93451a0>] submit_bio+0x70/0x150
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffc102f777>] ? lprocfs_oh_tally+0x17/0x40 [obdclass]
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffc16e1bdc>] osd_submit_bio+0x1c/0x60 [osd_ldiskfs]
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffc16e3d40>] osd_do_bio.isra.35+0x4f0/0xb40 [osd_ldiskfs]
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffc15d382b>] ? __ldiskfs_handle_dirty_metadata+0x8b/0x220 [ldiskfs]
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffa927882d>] ? __brelse+0x3d/0x50
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffc16e4b7c>] osd_write_commit+0x3ec/0x8c0 [osd_ldiskfs]
Apr 22 19:05:01 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:05:02 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:05:02 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:02 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:02 fir-io2-s2 kernel: Lustre: 84927:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff90f3dfb19c50 x1629293247198160/t0(0) o400->9f028a15-de05-fb03-e955-0f77d593ba1f@10.9.107.55@o2ib4:322/0 lens 224/0 e 1 to 0 dl 1555985107 ref 2 fl New:H/0/ffffffff rc 0/-1
Apr 22 19:05:02 fir-io2-s2 kernel: Lustre: 84927:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 11982 previous similar messages
Apr 22 19:05:02 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:02 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:02 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:02 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:03 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:03 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:03 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:03 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:03 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:03 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:03 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:03 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:03 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:04 fir-io2-s2 kernel: ll_ost03_099    S ffff90febaa08000     0 85204      2 0x00000080
Apr 22 19:05:04 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffa976a175>] ? _raw_spin_unlock_irqrestore+0x15/0x20
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:05:04 fir-io2-s2 kernel: Pid: 84756, comm: ll_ost00_034 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 19:05:04 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:04 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 19:05:04 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985104.84756
Apr 22 19:05:05 fir-io2-s2 kernel: Pid: 83760, comm: ll_ost00_017 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 19:05:05 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:05 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:05:05 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:05:05 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:05 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:05 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 19:05:05 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:06 fir-io2-s2 kernel: ll_ost_io01_054 D ffff90febaa09040     0 85205      2 0x00000080
Apr 22 19:05:06 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:06 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:06 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:06 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:06 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:06 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:06 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:05:06 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:07 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:08 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:08 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:08 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:08 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:08 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:08 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:08 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:08 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:08 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:09 fir-io2-s2 kernel: ll_ost_io00_042 D ffff90febaa0a080     0 85206      2 0x00000080
Apr 22 19:05:09 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:05:09 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:05:10 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:10 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:10 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:10 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:10 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:10 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:10 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:10 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:10 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:10 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:10 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:10 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:11 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:11 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:11 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:11 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:11 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:11 fir-io2-s2 kernel: ll_ost_io01_055 D ffff90febaa0b0c0     0 85207      2 0x00000080
Apr 22 19:05:11 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:11 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:05:11 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:05:11 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:05:11 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:05:11 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:05:12 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:05:13 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:13 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:13 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:13 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:13 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:13 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:13 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:13 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:13 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:13 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:13 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:13 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:14 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:14 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:14 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:14 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:14 fir-io2-s2 kernel: ll_ost_io03_024 D ffff90febaa0c100     0 85208      2 0x00000080
Apr 22 19:05:14 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:14 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:14 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:14 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:14 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:15 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:16 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:17 fir-io2-s2 kernel: ll_ost_io00_043 D ffff90febaa0d140     0 85209      2 0x00000080
Apr 22 19:05:17 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:17 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:05:17 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:05:17 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:17 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:05:17 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:05:17 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:17 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:17 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:19 fir-io2-s2 kernel: ll_ost_io01_056 D ffff90febaa0e180     0 85210      2 0x00000080
Apr 22 19:05:19 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:19 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:19 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:19 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:19 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:19 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:05:19 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:19 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:19 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:20 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:21 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:21 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:21 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:21 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:21 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:21 fir-io2-s2 kernel: ll_ost_io03_025 D ffff90feb81b0000     0 85211      2 0x00000080
Apr 22 19:05:21 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:21 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:21 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:21 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:22 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:22 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:22 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:05:22 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:22 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:22 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:22 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:05:22 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:05:22 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:22 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:24 fir-io2-s2 kernel: ll_ost_io01_057 D ffff90feb81b1040     0 85212      2 0x00000080
Apr 22 19:05:24 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:24 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:25 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:25 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:25 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:25 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:25 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:25 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:25 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:25 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:25 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:25 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:25 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:26 fir-io2-s2 kernel: ll_ost_io00_044 D ffff90feb81b2080     0 85213      2 0x00000080
Apr 22 19:05:26 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:26 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:26 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:26 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:26 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:26 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:05:26 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffa976a175>] ? _raw_spin_unlock_irqrestore+0x15/0x20
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:28 fir-io2-s2 kernel: ll_ost_io01_058 D ffff90feb81b30c0     0 85214      2 0x00000080
Apr 22 19:05:28 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:28 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:05:28 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:05:28 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:28 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:05:28 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:05:28 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:29 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:29 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:29 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:29 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:29 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:29 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:29 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:29 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:29 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:29 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:29 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:29 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:30 fir-io2-s2 kernel: ll_ost_io01_059 D ffff90feb81b4100     0 85215      2 0x00000080
Apr 22 19:05:30 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:05:30 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:31 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:31 fir-io2-s2 kernel: LNet: Service thread pid 84968 was inactive for 555.71s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Apr 22 19:05:31 fir-io2-s2 kernel: LNet: Skipped 377 previous similar messages
Apr 22 19:05:31 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985131.84968
Apr 22 19:05:31 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:31 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:31 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:31 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:05:31 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:05:31 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:31 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:33 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:33 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:33 fir-io2-s2 kernel: ll_ost_io02_050 D ffff90feb81b5140     0 85216      2 0x00000080
Apr 22 19:05:33 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:33 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:33 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:33 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:33 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:05:33 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:34 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:35 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:35 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:35 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:35 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:35 fir-io2-s2 kernel: ll_ost_io00_045 D ffff90feb81b6180     0 85217      2 0x00000080
Apr 22 19:05:35 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:36 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:36 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:36 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:36 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:36 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:05:36 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:36 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:36 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:36 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:05:36 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:38 fir-io2-s2 kernel: ll_ost_io00_046 D ffff910091068000     0 85218      2 0x00000080
Apr 22 19:05:38 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:38 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:05:38 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:05:38 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:05:38 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:05:38 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:05:38 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:05:38 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:05:38 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:05:39 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:05:39 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:05:39 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:05:39 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:05:39 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:39 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985139.84929
Apr 22 19:05:39 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:39 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:05:39 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:05:39 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:05:39 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:39 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:41 fir-io2-s2 kernel: ll_ost_io00_047 D ffff910091069040     0 85219      2 0x00000080
Apr 22 19:05:41 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:41 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:41 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:41 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:41 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:41 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:41 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:05:41 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:41 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:41 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:42 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:42 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:42 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:05:42 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:42 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:42 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:42 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:42 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:42 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:42 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:42 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:43 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:43 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:43 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:43 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:43 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:43 fir-io2-s2 kernel: ll_ost_io01_060 D ffff91009106a080     0 85220      2 0x00000080
Apr 22 19:05:43 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:43 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 19:05:43 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 19:05:43 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffc16763c3>] ? qsd_acquire+0x8e3/0xcb0 [lquota]
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:05:44 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:05:45 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:05:45 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:05:45 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:05:45 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:45 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:45 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:45 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:45 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:45 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:45 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:45 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:45 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:46 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:46 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:46 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:05:46 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:46 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:46 fir-io2-s2 kernel: ll_ost_io02_051 D ffff91009106b0c0     0 85221      2 0x00000080
Apr 22 19:05:46 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:46 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:46 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:47 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:47 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:47 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:05:47 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:47 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:47 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:47 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:47 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:47 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:05:47 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:47 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:47 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:48 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:48 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:48 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:48 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:48 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:48 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:48 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:48 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:48 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:48 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:48 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:48 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:49 fir-io2-s2 kernel: ll_ost_io00_048 D ffff91009106c100     0 85222      2 0x00000080
Apr 22 19:05:49 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:50 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:50 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:50 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:05:50 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 19:05:50 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:05:50 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:50 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:50 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:50 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:50 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:50 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:50 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:51 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:51 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:51 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:51 fir-io2-s2 kernel: ll_ost_io01_061 D ffff91009106d140     0 85223      2 0x00000080
Apr 22 19:05:51 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:51 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:51 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:51 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:51 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:51 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:05:51 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:52 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:52 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:52 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:52 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:52 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:05:52 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:05:52 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:52 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:52 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:52 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:52 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:52 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:54 fir-io2-s2 kernel: ll_ost_io02_052 D ffff91009106e180     0 85224      2 0x00000080
Apr 22 19:05:54 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa937c8b4>] ? __radix_tree_lookup+0x84/0xf0
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa9765b71>] out_of_line_wait_on_bit_lock+0x81/0xb0
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:05:54 fir-io2-s2 kernel: [<ffffffffa9278f42>] __lock_buffer+0x32/0x40
Apr 22 19:05:55 fir-io2-s2 kernel: [<ffffffffc09e1ea2>] do_get_write_access+0x432/0x4d0 [jbd2]
Apr 22 19:05:55 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 19:05:55 fir-io2-s2 kernel: [<ffffffffa90a9aad>] ? internal_add_timer+0x7d/0xc0
Apr 22 19:05:55 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:05:55 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:05:55 fir-io2-s2 kernel: [<ffffffffc15e1c7d>] ldiskfs_mb_mark_diskspace_used+0x7d/0x510 [ldiskfs]
Apr 22 19:05:55 fir-io2-s2 kernel: [<ffffffffc15e5bb0>] ldiskfs_mb_new_blocks+0x350/0xb20 [ldiskfs]
Apr 22 19:05:55 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:05:55 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:05:55 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:05:55 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:05:55 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:56 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:57 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:57 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:57 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:05:57 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:57 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:05:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:05:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:05:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:05:57 fir-io2-s2 kernel: ll_ost_io00_049 D ffff90fea7d58000     0 85225      2 0x00000080
Apr 22 19:05:57 fir-io2-s2 kernel: Call Trace:
Apr 22 19:05:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:05:58 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:05:59 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:00 fir-io2-s2 kernel: ll_ost_io03_026 D ffff90fea7d59040     0 85226      2 0x00000080
Apr 22 19:06:00 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:00 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:06:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:00 fir-io2-s2 kernel: [<ffffffffa9769535>] rwsem_down_write_failed+0x225/0x3a0
Apr 22 19:06:00 fir-io2-s2 kernel: [<ffffffffa9386c47>] call_rwsem_down_write_failed+0x17/0x30
Apr 22 19:06:00 fir-io2-s2 kernel: [<ffffffffa9766f4d>] down_write+0x2d/0x3d
Apr 22 19:06:00 fir-io2-s2 kernel: [<ffffffffc15ad5b4>] ldiskfs_map_blocks+0x134/0x700 [ldiskfs]
Apr 22 19:06:00 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:06:01 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:06:01 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:01 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:06:01 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:01 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:01 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:01 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:01 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:01 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:01 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:01 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:01 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:02 fir-io2-s2 kernel: ll_ost_io01_062 D ffff90e2b99d2080     0 85227      2 0x00000080
Apr 22 19:06:02 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:06:02 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:06:03 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:06:03 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:06:03 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:06:03 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:03 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:03 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:03 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:03 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:03 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:03 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:03 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:03 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:04 fir-io2-s2 kernel: ll_ost_io02_053 D ffff90e2b99d4100     0 85228      2 0x00000080
Apr 22 19:06:04 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:06:04 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:06:05 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:06:05 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:06:05 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:06:05 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:06:05 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:06:05 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:06:05 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 19:06:05 fir-io2-s2 kernel: [<ffffffffc16615a2>] ? lquota_disk_read+0xf2/0x390 [lquota]
Apr 22 19:06:05 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:06:05 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:06:05 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:06:05 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:06:06 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:06:06 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:06:06 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:06:06 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:06:06 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:06 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:06:06 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:06 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:06 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:06 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:06 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:06 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:08 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985167.84758
Apr 22 19:06:08 fir-io2-s2 kernel: ll_ost_io01_063 D ffff90ff8e0d0000     0 85229      2 0x00000080
Apr 22 19:06:08 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:08 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:06:08 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:06:08 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:06:08 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:06:08 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:06:08 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:06:08 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:06:08 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:06:08 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:06:08 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:06:08 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:06:09 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:06:09 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:06:09 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:06:09 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:06:09 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:06:09 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:06:09 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:09 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:06:09 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:09 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:09 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:09 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:11 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:11 fir-io2-s2 kernel: ll_ost_io01_064 D ffff90ff8e0d1040     0 85230      2 0x00000080
Apr 22 19:06:11 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:11 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:11 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:06:11 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:06:11 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:06:11 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:06:11 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:06:11 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:06:11 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:06:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:06:12 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985172.84800
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:13 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:13 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:13 fir-io2-s2 kernel: ll_ost_io01_065 D ffff90ff8e0d2080     0 85231      2 0x00000080
Apr 22 19:06:13 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:13 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:06:13 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:06:13 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:06:13 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:06:13 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:06:13 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:06:14 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:06:14 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:06:14 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985174.84880
Apr 22 19:06:14 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:06:14 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:06:14 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:06:14 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:06:14 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:14 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:14 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:14 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:14 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:14 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:16 fir-io2-s2 kernel: ll_ost_io03_027 D ffff90ff8e0d30c0     0 85232      2 0x00000080
Apr 22 19:06:16 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:16 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:06:16 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:06:16 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:06:16 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:06:16 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:06:16 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:06:16 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:06:16 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:06:16 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:06:16 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:06:16 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:06:17 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:06:17 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:06:17 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:06:17 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:06:17 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:17 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:06:17 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:17 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:17 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:17 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:17 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:17 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:19 fir-io2-s2 kernel: ll_ost_io03_028 D ffff90ff8e0d4100     0 85233      2 0x00000080
Apr 22 19:06:19 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:19 fir-io2-s2 kernel: [<ffffffffc16763c3>] ? qsd_acquire+0x8e3/0xcb0 [lquota]
Apr 22 19:06:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:19 fir-io2-s2 kernel: [<ffffffffa9769535>] rwsem_down_write_failed+0x225/0x3a0
Apr 22 19:06:19 fir-io2-s2 kernel: [<ffffffffa921a849>] ? ___slab_alloc+0x209/0x4f0
Apr 22 19:06:19 fir-io2-s2 kernel: [<ffffffffa9386c47>] call_rwsem_down_write_failed+0x17/0x30
Apr 22 19:06:19 fir-io2-s2 kernel: [<ffffffffa9766f4d>] down_write+0x2d/0x3d
Apr 22 19:06:19 fir-io2-s2 kernel: [<ffffffffc15ad5b4>] ldiskfs_map_blocks+0x134/0x700 [ldiskfs]
Apr 22 19:06:19 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:06:19 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:06:19 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:19 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:06:19 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:21 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:21 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:21 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:21 fir-io2-s2 kernel: ll_ost_io01_066 D ffff90ff8e0d5140     0 85234      2 0x00000080
Apr 22 19:06:21 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:21 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:06:21 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:06:21 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:06:21 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:06:21 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:06:21 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:06:21 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:06:22 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:06:22 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:06:22 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:06:22 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:06:22 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:06:22 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985182.86636
Apr 22 19:06:22 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:06:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:06:22 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:06:22 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:22 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:06:22 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:24 fir-io2-s2 kernel: ll_ost_io00_050 D ffff90ff8e0d6180     0 85235      2 0x00000080
Apr 22 19:06:24 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:24 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:06:24 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:06:24 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:06:24 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:06:24 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:06:24 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:25 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:26 fir-io2-s2 kernel: ll_ost_io02_054 D ffff90febaaa0000     0 85236      2 0x00000080
Apr 22 19:06:26 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:26 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:06:26 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:06:26 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:06:26 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:06:26 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:06:26 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:26 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:28 fir-io2-s2 kernel: ll_ost_io02_055 S ffff90febaaa1040     0 85237      2 0x00000080
Apr 22 19:06:28 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:28 fir-io2-s2 kernel: [<ffffffffc1248805>] ptlrpc_wait_event+0x345/0x360 [ptlrpc]
Apr 22 19:06:28 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:06:28 fir-io2-s2 kernel: [<ffffffffc124ed63>] ptlrpc_main+0x853/0x1fc0 [ptlrpc]
Apr 22 19:06:28 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:28 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:28 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:28 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:29 fir-io2-s2 kernel: ll_ost_io00_051 D ffff90febaaa2080     0 85238      2 0x00000080
Apr 22 19:06:29 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:29 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:29 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:06:29 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:06:29 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:06:29 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:06:29 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:06:29 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:06:29 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:06:29 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:06:29 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:06:29 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:06:30 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:06:30 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:30 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:30 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:30 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:30 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:30 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:30 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:30 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:30 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:30 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:30 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:31 fir-io2-s2 kernel: ll_ost_io01_067 D ffff90febaaa30c0     0 85239      2 0x00000080
Apr 22 19:06:31 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffa959a485>] bitmap_startwrite+0x1f5/0x210
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffc0bcf2e1>] add_stripe_bio+0x421/0x7d0 [raid456]
Apr 22 19:06:31 fir-io2-s2 kernel: [<ffffffffc0bd4c34>] raid5_make_request+0x1e4/0xc70 [raid456]
Apr 22 19:06:32 fir-io2-s2 kernel: [<ffffffffc15ecd00>] ? __ldiskfs_ext_dirty+0x40/0xa0 [ldiskfs]
Apr 22 19:06:32 fir-io2-s2 kernel: [<ffffffffc15ecd33>] ? __ldiskfs_ext_dirty+0x73/0xa0 [ldiskfs]
Apr 22 19:06:32 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:06:32 fir-io2-s2 kernel: [<ffffffffa958d330>] md_handle_request+0xd0/0x150
Apr 22 19:06:32 fir-io2-s2 kernel: [<ffffffffa934311f>] ? generic_make_request_checks+0x27f/0x390
Apr 22 19:06:32 fir-io2-s2 kernel: [<ffffffffa958d499>] md_make_request+0x79/0x190
Apr 22 19:06:32 fir-io2-s2 kernel: [<ffffffffa9344ef7>] generic_make_request+0x147/0x380
Apr 22 19:06:32 fir-io2-s2 kernel: [<ffffffffa9587a86>] ? md_mergeable_bvec+0x46/0x50
Apr 22 19:06:32 fir-io2-s2 kernel: [<ffffffffa93451a0>] submit_bio+0x70/0x150
Apr 22 19:06:32 fir-io2-s2 kernel: [<ffffffffc102f777>] ? lprocfs_oh_tally+0x17/0x40 [obdclass]
Apr 22 19:06:32 fir-io2-s2 kernel: [<ffffffffc16e1bdc>] osd_submit_bio+0x1c/0x60 [osd_ldiskfs]
Apr 22 19:06:32 fir-io2-s2 kernel: [<ffffffffc16e3d40>] osd_do_bio.isra.35+0x4f0/0xb40 [osd_ldiskfs]
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffc15d382b>] ? __ldiskfs_handle_dirty_metadata+0x8b/0x220 [ldiskfs]
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffa927882d>] ? __brelse+0x3d/0x50
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffc16e4b7c>] osd_write_commit+0x3ec/0x8c0 [osd_ldiskfs]
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:33 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:34 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:34 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:34 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:34 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:34 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:34 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:34 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:34 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:34 fir-io2-s2 kernel: ll_ost_io00_052 D ffff90febaaa4100     0 85240      2 0x00000080
Apr 22 19:06:34 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:34 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:06:35 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:06:36 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:36 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:06:36 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:36 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:36 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:36 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:36 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:36 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:36 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:36 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:36 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:36 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:37 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:37 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:37 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:37 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:37 fir-io2-s2 kernel: ll_ost_io00_053 D ffff90febaaa5140     0 85241      2 0x00000080
Apr 22 19:06:37 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:37 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:06:37 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:06:37 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:06:38 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:06:38 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:06:38 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:06:38 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:06:38 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:06:38 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:06:38 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:06:38 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:06:38 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:06:38 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:06:38 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:39 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:40 fir-io2-s2 kernel: ll_ost_io02_056 D ffff90febaaa6180     0 85242      2 0x00000080
Apr 22 19:06:40 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:06:40 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:06:41 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:06:41 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:06:41 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:06:41 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:41 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:41 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:41 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:41 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:41 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:41 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:41 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:41 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:42 fir-io2-s2 kernel: ll_ost_io02_057 D ffff90fea7d5a080     0 85243      2 0x00000080
Apr 22 19:06:42 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:06:42 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:06:43 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:06:43 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:43 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:06:43 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:06:43 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:06:43 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:06:43 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:06:43 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:06:43 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:06:43 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:06:43 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:43 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:44 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:44 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:44 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:44 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:44 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:44 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:44 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:44 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:44 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:44 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:44 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:44 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:45 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:45 fir-io2-s2 kernel: ll_ost_io00_054 D ffff90fea7d5b0c0     0 85244      2 0x00000080
Apr 22 19:06:45 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:45 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:45 fir-io2-s2 kernel: [<ffffffffa959a485>] bitmap_startwrite+0x1f5/0x210
Apr 22 19:06:45 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:06:45 fir-io2-s2 kernel: [<ffffffffc0bcf2e1>] add_stripe_bio+0x421/0x7d0 [raid456]
Apr 22 19:06:45 fir-io2-s2 kernel: [<ffffffffc0bd4c34>] raid5_make_request+0x1e4/0xc70 [raid456]
Apr 22 19:06:45 fir-io2-s2 kernel: [<ffffffffc15ecd00>] ? __ldiskfs_ext_dirty+0x40/0xa0 [ldiskfs]
Apr 22 19:06:45 fir-io2-s2 kernel: [<ffffffffc15ecd33>] ? __ldiskfs_ext_dirty+0x73/0xa0 [ldiskfs]
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffa958d330>] md_handle_request+0xd0/0x150
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffa934311f>] ? generic_make_request_checks+0x27f/0x390
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffa958d499>] md_make_request+0x79/0x190
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffa9344ef7>] generic_make_request+0x147/0x380
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffa9587a86>] ? md_mergeable_bvec+0x46/0x50
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffa93451a0>] submit_bio+0x70/0x150
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffc102f777>] ? lprocfs_oh_tally+0x17/0x40 [obdclass]
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffc16e1bdc>] osd_submit_bio+0x1c/0x60 [osd_ldiskfs]
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffc16e3d40>] osd_do_bio.isra.35+0x4f0/0xb40 [osd_ldiskfs]
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffc15d382b>] ? __ldiskfs_handle_dirty_metadata+0x8b/0x220 [ldiskfs]
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffa927882d>] ? __brelse+0x3d/0x50
Apr 22 19:06:46 fir-io2-s2 kernel: [<ffffffffc16e4b7c>] osd_write_commit+0x3ec/0x8c0 [osd_ldiskfs]
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:47 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:48 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:48 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:48 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:48 fir-io2-s2 kernel: ll_ost_io03_029 D ffff90fea7d5c100     0 85245      2 0x00000080
Apr 22 19:06:48 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:48 fir-io2-s2 kernel: [<ffffffffa921a849>] ? ___slab_alloc+0x209/0x4f0
Apr 22 19:06:48 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:06:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:48 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:06:48 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985208.84844
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffa93667ca>] ? blk_throtl_bio+0xda/0x510
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffc0bd419d>] ? raid5_compute_blocknr+0xbd/0x290 [raid456]
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffa93667ca>] ? blk_throtl_bio+0xda/0x510
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 19:06:49 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 19:06:50 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:06:50 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:06:50 fir-io2-s2 kernel: [<ffffffffc15cd6a6>] ldiskfs_quota_write+0xc6/0x1e0 [ldiskfs]
Apr 22 19:06:50 fir-io2-s2 kernel: [<ffffffffa92b2a95>] qtree_write_dquot+0xd5/0x180
Apr 22 19:06:50 fir-io2-s2 kernel: [<ffffffffa92b184b>] v2_write_dquot+0x2b/0x30
Apr 22 19:06:50 fir-io2-s2 kernel: [<ffffffffa92ae36d>] dquot_commit+0xbd/0xd0
Apr 22 19:06:50 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:06:50 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:06:50 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:06:50 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:06:50 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:06:50 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:06:51 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:06:51 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:06:51 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:06:51 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:06:51 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:06:51 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:06:51 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:51 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:06:51 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:51 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:51 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:51 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:53 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985213.84798
Apr 22 19:06:53 fir-io2-s2 kernel: ll_ost_io03_030 D ffff90fea7d5d140     0 85246      2 0x00000080
Apr 22 19:06:53 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:53 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:06:53 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:06:53 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:06:53 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:06:53 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:06:53 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:06:53 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:06:53 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:06:53 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:06:54 fir-io2-s2 kernel: [<ffffffffc15de03b>] ? ldiskfs_es_insert_extent+0xdb/0x1c0 [ldiskfs]
Apr 22 19:06:54 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:06:54 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:06:54 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:06:54 fir-io2-s2 kernel: [<ffffffffc15ebcd3>] ? ldiskfs_ext_put_gap_in_cache+0x83/0x90 [ldiskfs]
Apr 22 19:06:54 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:06:54 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:06:54 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:06:54 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:06:54 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:54 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:06:54 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:55 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:55 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:55 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:55 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:55 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:55 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:55 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:55 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:55 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:55 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:55 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:55 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:56 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:56 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:56 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:56 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:56 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:56 fir-io2-s2 kernel: ll_ost_io01_068 D ffff90fea7d5e180     0 85247      2 0x00000080
Apr 22 19:06:56 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:56 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 19:06:56 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 19:06:56 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:06:56 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:06:56 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:06:56 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffc16763c3>] ? qsd_acquire+0x8e3/0xcb0 [lquota]
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:06:57 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:06:58 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:06:58 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:06:58 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:06:58 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:06:58 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:06:58 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:06:58 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:06:58 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:06:58 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:06:58 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:06:58 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:06:58 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:06:59 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:06:59 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:06:59 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:06:59 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:06:59 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:06:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:06:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:06:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:06:59 fir-io2-s2 kernel: ll_ost_io00_055 D ffff90feacf10000     0 85248      2 0x00000080
Apr 22 19:06:59 fir-io2-s2 kernel: Call Trace:
Apr 22 19:06:59 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:06:59 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:06:59 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:00 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:01 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:01 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:01 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:01 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:01 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:01 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:01 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:01 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:01 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:01 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:01 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:01 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:02 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:02 fir-io2-s2 kernel: ll_ost_io03_031 D ffff90feacf11040     0 85249      2 0x00000080
Apr 22 19:07:02 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:02 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:07:02 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:02 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:07:02 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:07:02 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:07:02 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:07:02 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:02 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:03 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:07:03 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:03 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:07:03 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:07:03 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:03 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:03 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:03 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:03 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:03 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:03 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:03 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:04 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:04 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:04 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:04 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:04 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:04 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:04 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:04 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:04 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:04 fir-io2-s2 kernel: ll_ost_io02_058 D ffff90feacf12080     0 85250      2 0x00000080
Apr 22 19:07:04 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:04 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:07:05 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:07:05 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:07:05 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:07:05 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:07:05 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:07:05 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:07:05 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:07:05 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:07:05 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:07:05 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:07:05 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:07:05 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:06 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:07 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:07 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:07 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:07 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:07 fir-io2-s2 kernel: ll_ost_io00_056 D ffff90feacf130c0     0 85251      2 0x00000080
Apr 22 19:07:07 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:07 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:07 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:07:07 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:08 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:09 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:09 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:09 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:09 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:09 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:09 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:09 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:09 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:09 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:09 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:09 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:10 fir-io2-s2 kernel: ll_ost_io00_057 D ffff90feacf14100     0 85252      2 0x00000080
Apr 22 19:07:10 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:10 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:07:10 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:10 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:07:10 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:07:10 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:07:10 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:11 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:12 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:12 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:12 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:12 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:12 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:13 fir-io2-s2 kernel: ll_ost_io02_059 D ffff90feacf15140     0 85253      2 0x00000080
Apr 22 19:07:13 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:07:13 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:07:14 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:14 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:14 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:14 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:14 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:14 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:14 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:14 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:14 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:14 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:14 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:15 fir-io2-s2 kernel: ll_ost_io00_058 D ffff90feacf16180     0 85254      2 0x00000080
Apr 22 19:07:15 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:15 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:07:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:15 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:07:15 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:16 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:17 fir-io2-s2 kernel: ll_ost_io02_060 D ffff910219d48000     0 85255      2 0x00000080
Apr 22 19:07:17 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:17 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:07:17 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:07:17 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:07:17 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:07:18 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:07:18 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:07:18 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:07:18 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:07:18 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:07:18 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 19:07:18 fir-io2-s2 kernel: [<ffffffffc16615a2>] ? lquota_disk_read+0xf2/0x390 [lquota]
Apr 22 19:07:18 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:07:18 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:07:18 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:07:18 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:07:18 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:07:19 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:07:19 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:07:19 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:07:19 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:07:19 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:07:19 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:19 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:19 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:19 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:19 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:19 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:19 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:20 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:20 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:20 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:20 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:20 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:20 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:20 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:21 fir-io2-s2 kernel: ll_ost_io00_059 D ffff910219d49040     0 85256      2 0x00000080
Apr 22 19:07:21 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:07:21 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:22 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:22 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:07:22 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:07:22 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:07:22 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:22 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:22 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:22 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:22 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:22 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:22 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:22 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:23 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:23 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:23 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:23 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:23 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:23 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:23 fir-io2-s2 kernel: ll_ost_io00_060 D ffff910219d4a080     0 85257      2 0x00000080
Apr 22 19:07:23 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:24 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:25 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:25 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:25 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:25 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:25 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:25 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:25 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:25 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:25 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:25 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:25 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:25 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:26 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:26 fir-io2-s2 kernel: ll_ost_io03_032 D ffff90fedea68000     0 85258      2 0x00000080
Apr 22 19:07:26 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:26 fir-io2-s2 kernel: [<ffffffffa927e555>] ? bio_alloc_bioset+0x115/0x310
Apr 22 19:07:26 fir-io2-s2 kernel: [<ffffffffa90dc766>] ? select_task_rq_fair+0x5a6/0x760
Apr 22 19:07:26 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:07:26 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:26 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:07:26 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 19:07:26 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:07:27 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 19:07:28 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 19:07:28 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 19:07:28 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 19:07:28 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 19:07:28 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 19:07:28 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 19:07:28 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:07:28 fir-io2-s2 kernel: [<ffffffffc16c6699>] ? osd_attr_set+0x199/0xb90 [osd_ldiskfs]
Apr 22 19:07:28 fir-io2-s2 kernel: [<ffffffffc16e759c>] ? osd_punch+0x28c/0x6b0 [osd_ldiskfs]
Apr 22 19:07:28 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:07:28 fir-io2-s2 kernel: [<ffffffffc180892b>] ofd_object_punch+0x4cb/0xd30 [ofd]
Apr 22 19:07:28 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:07:29 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:29 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:29 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:29 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:29 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:29 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:29 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:29 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985249.84792
Apr 22 19:07:30 fir-io2-s2 kernel: ll_ost_io01_069 D ffff90fedea69040     0 85259      2 0x00000080
Apr 22 19:07:30 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:30 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:31 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:31 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:31 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:31 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:31 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:31 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:31 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:31 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:31 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:31 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:31 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:32 fir-io2-s2 kernel: ll_ost_io01_070 D ffff90fedea6a080     0 85260      2 0x00000080
Apr 22 19:07:32 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:32 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:07:32 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:07:32 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:07:32 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:07:32 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:07:32 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:07:32 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:07:32 fir-io2-s2 kernel: [<ffffffffa92ae6cc>] ? dqput+0x4c/0x1f0
Apr 22 19:07:32 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:07:32 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 19:07:32 fir-io2-s2 kernel: [<ffffffffc16615a2>] ? lquota_disk_read+0xf2/0x390 [lquota]
Apr 22 19:07:33 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:07:33 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:07:33 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:07:33 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:07:33 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:07:33 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:07:33 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:07:33 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:07:33 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:07:33 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:07:33 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:33 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:34 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985253.84974
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:34 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:35 fir-io2-s2 kernel: ll_ost_io03_033 D ffff90fedea6b0c0     0 85261      2 0x00000080
Apr 22 19:07:35 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:35 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:07:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:35 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:07:35 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:07:35 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:07:35 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:07:36 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:36 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985256.82758
Apr 22 19:07:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:36 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:07:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:36 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:07:36 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:36 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:36 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:36 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:36 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:36 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:38 fir-io2-s2 kernel: ll_ost_io02_061 D ffff90fedea6c100     0 85262      2 0x00000080
Apr 22 19:07:38 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa921a849>] ? ___slab_alloc+0x209/0x4f0
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa9394d0d>] ? list_del+0xd/0x30
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa921a07e>] ? get_partial_node.isra.49+0xfe/0x230
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa93667ca>] ? blk_throtl_bio+0xda/0x510
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:07:38 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffc15cd6a6>] ldiskfs_quota_write+0xc6/0x1e0 [ldiskfs]
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffa92b2a95>] qtree_write_dquot+0xd5/0x180
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffa92b184b>] v2_write_dquot+0x2b/0x30
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffa92ae36d>] dquot_commit+0xbd/0xd0
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffa90a9aad>] ? internal_add_timer+0x7d/0xc0
Apr 22 19:07:39 fir-io2-s2 kernel: [<ffffffffa90aad4c>] ? mod_timer+0x10c/0x230
Apr 22 19:07:40 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:07:40 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:07:40 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:07:40 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:07:40 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:40 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:40 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:07:40 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:07:40 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:07:40 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:40 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:40 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:41 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:42 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:42 fir-io2-s2 kernel: ll_ost_io00_061 D ffff90fedea6d140     0 85263      2 0x00000080
Apr 22 19:07:42 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:42 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:42 fir-io2-s2 kernel: [<ffffffffa9769535>] rwsem_down_write_failed+0x225/0x3a0
Apr 22 19:07:42 fir-io2-s2 kernel: [<ffffffffa9386c47>] call_rwsem_down_write_failed+0x17/0x30
Apr 22 19:07:42 fir-io2-s2 kernel: [<ffffffffa9766f4d>] down_write+0x2d/0x3d
Apr 22 19:07:42 fir-io2-s2 kernel: [<ffffffffc15ad5b4>] ldiskfs_map_blocks+0x134/0x700 [ldiskfs]
Apr 22 19:07:42 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:07:42 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:07:42 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:07:42 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:43 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:44 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:44 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:44 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:44 fir-io2-s2 kernel: ll_ost_io01_071 D ffff90fedea6e180     0 85264      2 0x00000080
Apr 22 19:07:44 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:44 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:07:44 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:07:44 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:44 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:07:44 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:07:44 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:45 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:45 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:45 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:45 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:45 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:45 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:45 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:45 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:45 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:45 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:45 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:45 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:46 fir-io2-s2 kernel: ll_ost_io00_062 D ffff90fea5240000     0 85265      2 0x00000080
Apr 22 19:07:46 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffa93210f2>] ? crypto_create_tfm+0x32/0xd0
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:07:46 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:07:47 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:07:47 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:47 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:47 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:07:47 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:47 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:07:47 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:47 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:47 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:47 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:47 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:47 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:48 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:48 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:48 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:48 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:48 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:48 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:48 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:48 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:48 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:49 fir-io2-s2 kernel: ll_ost_io00_063 D ffff90fea5241040     0 85266      2 0x00000080
Apr 22 19:07:49 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:49 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:49 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:07:49 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:49 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:07:49 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:07:49 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:07:49 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:07:49 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:49 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:07:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:51 fir-io2-s2 kernel: ll_ost_io03_034 D ffff90fea5242080     0 85267      2 0x00000080
Apr 22 19:07:51 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:51 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:07:51 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:07:51 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:51 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:07:51 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:07:51 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:51 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:51 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:51 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:52 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:52 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:52 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:52 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:52 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:52 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:52 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:52 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:52 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:52 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:52 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:53 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:53 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:53 fir-io2-s2 kernel: ll_ost_io00_064 D ffff90fea52430c0     0 85268      2 0x00000080
Apr 22 19:07:53 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:53 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:07:53 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:07:53 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:53 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:07:53 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:07:53 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:53 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:53 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:55 fir-io2-s2 kernel: ll_ost_io01_072 D ffff90fea5244100     0 85269      2 0x00000080
Apr 22 19:07:55 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:55 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:07:55 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:55 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:07:55 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:07:55 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:07:55 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:07:55 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:55 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:56 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:07:56 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:56 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:07:56 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:56 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:56 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:56 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:56 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:07:56 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:07:56 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:56 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:56 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:07:57 fir-io2-s2 kernel: ll_ost_io00_065 D ffff90fea5245140     0 85270      2 0x00000080
Apr 22 19:07:57 fir-io2-s2 kernel: Call Trace:
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:07:57 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:07:58 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:07:58 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:07:58 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:07:58 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:07:58 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:07:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:58 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:07:58 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:07:58 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:07:58 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:07:58 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:07:58 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:07:59 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:07:59 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:07:59 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:07:59 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:07:59 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:07:59 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:07:59 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:07:59 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:07:59 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:07:59 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:07:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:07:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:00 fir-io2-s2 kernel: ll_ost_io03_035 D ffff90fea5246180     0 85271      2 0x00000080
Apr 22 19:08:00 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:08:00 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985280.84985
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:08:00 fir-io2-s2 kernel: [<ffffffffa9765b71>] out_of_line_wait_on_bit_lock+0x81/0xb0
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffa9278f42>] __lock_buffer+0x32/0x40
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffc09e1ea2>] do_get_write_access+0x432/0x4d0 [jbd2]
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 19:08:01 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 19:08:02 fir-io2-s2 kernel: [<ffffffffa9269218>] ? generic_setxattr+0x68/0x80
Apr 22 19:08:02 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 19:08:02 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 19:08:02 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:08:02 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:08:02 fir-io2-s2 kernel: [<ffffffffc180892b>] ofd_object_punch+0x4cb/0xd30 [ofd]
Apr 22 19:08:02 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:08:02 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:02 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985282.84886
Apr 22 19:08:02 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:02 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:03 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:03 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:03 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:03 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:03 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:03 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:03 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:03 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:03 fir-io2-s2 kernel: ll_ost_io00_066 D ffff90feddc20000     0 85272      2 0x00000080
Apr 22 19:08:03 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:03 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:08:03 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:08:03 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:04 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985284.85166
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:08:04 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:08:05 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:08:05 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:05 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:05 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:05 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:05 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:05 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:05 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:05 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:05 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:05 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:05 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:06 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:06 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:06 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:06 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:06 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:06 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:06 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:06 fir-io2-s2 kernel: ll_ost_io03_036 D ffff910219d4b0c0     0 85273      2 0x00000080
Apr 22 19:08:06 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:06 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 19:08:06 fir-io2-s2 kernel: [<ffffffffc1036a51>] ? lprocfs_counter_sub+0xc1/0x130 [obdclass]
Apr 22 19:08:06 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffc122c3e0>] ? ptlrpc_req_finished+0x10/0x20 [ptlrpc]
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffc1668347>] ? qsd_send_dqacq+0x307/0x340 [lquota]
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:08:07 fir-io2-s2 kernel: [<ffffffffc16763c3>] ? qsd_acquire+0x8e3/0xcb0 [lquota]
Apr 22 19:08:08 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:08:08 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:08:08 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:08:08 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:08:08 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:08:08 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:08:08 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:08 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:08 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:08 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:08 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:08 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:09 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:09 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:09 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:09 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:09 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:09 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:09 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:09 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:10 fir-io2-s2 kernel: ll_ost_io00_067 D ffff910219d4c100     0 85274      2 0x00000080
Apr 22 19:08:10 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:10 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:08:10 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:08:10 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:08:10 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:08:10 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:08:10 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:08:10 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:10 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:10 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:08:10 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:11 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:08:11 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:11 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:11 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:11 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:11 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:11 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:11 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:11 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:11 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:11 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:11 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:12 fir-io2-s2 kernel: ll_ost_io02_062 D ffff910219d4d140     0 85275      2 0x00000080
Apr 22 19:08:12 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:08:12 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:08:13 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:08:13 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:08:13 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:08:13 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:08:13 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:08:13 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:08:13 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:08:13 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:08:13 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:08:13 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:08:13 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:08:13 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:14 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:15 fir-io2-s2 kernel: ll_ost_io03_037 D ffff910219d4e180     0 85276      2 0x00000080
Apr 22 19:08:15 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:15 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:08:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:08:15 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:08:15 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:08:16 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:08:16 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:08:16 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:16 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:08:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:16 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:08:16 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:08:16 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:16 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:16 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:16 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:18 fir-io2-s2 kernel: ll_ost_io02_063 D ffff90fec9370000     0 85277      2 0x00000080
Apr 22 19:08:18 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:18 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:08:18 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:08:18 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:08:18 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:08:18 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:08:18 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:08:18 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:18 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:18 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:08:18 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:19 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:20 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:20 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:20 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:20 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:20 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:20 fir-io2-s2 kernel: ll_ost_io03_038 D ffff90fec9371040     0 85278      2 0x00000080
Apr 22 19:08:20 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:20 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:08:20 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:08:20 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:08:20 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:08:20 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:08:21 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:08:21 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:08:21 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:08:21 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:08:21 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:08:21 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:08:21 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:08:21 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:21 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:21 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:08:21 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:08:21 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:22 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:23 fir-io2-s2 kernel: ll_ost_io00_068 D ffff90fec9372080     0 85279      2 0x00000080
Apr 22 19:08:23 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:08:23 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:08:24 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 19:08:24 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:08:24 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 19:08:24 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:08:24 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 19:08:24 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 19:08:24 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:08:24 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:08:24 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 19:08:24 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:08:24 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 19:08:24 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 19:08:25 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 19:08:25 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 19:08:25 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 19:08:25 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 19:08:25 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 19:08:25 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:08:25 fir-io2-s2 kernel: [<ffffffffc1036929>] ? lprocfs_counter_add+0xf9/0x160 [obdclass]
Apr 22 19:08:25 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:08:25 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:08:25 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:25 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:25 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:26 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:27 fir-io2-s2 kernel: ll_ost_io03_039 D ffff90fec93730c0     0 85280      2 0x00000080
Apr 22 19:08:27 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:27 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:08:27 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:08:27 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:08:27 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:08:27 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:08:27 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:08:27 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:27 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:27 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:08:28 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:28 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:08:28 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:28 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:28 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:28 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:28 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:28 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:28 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:28 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:28 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:28 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:29 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:29 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:29 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:29 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:29 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:29 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:29 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:29 fir-io2-s2 kernel: ll_ost_io03_040 D ffff90fec9374100     0 85281      2 0x00000080
Apr 22 19:08:29 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:29 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:08:29 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:08:29 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:08:30 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:31 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:32 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:32 fir-io2-s2 kernel: ll_ost_io02_064 D ffff90fec9375140     0 85282      2 0x00000080
Apr 22 19:08:32 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:32 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:08:32 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:08:32 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:08:32 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:08:32 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:08:33 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:08:33 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:08:33 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:08:33 fir-io2-s2 kernel: [<ffffffffa90e0000>] ? update_curr+0x30/0x1e0
Apr 22 19:08:33 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:08:33 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:08:33 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:08:33 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:08:33 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:33 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:33 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:08:33 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:34 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:35 fir-io2-s2 kernel: ll_ost_io03_041 D ffff90fec9376180     0 85283      2 0x00000080
Apr 22 19:08:35 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:35 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:08:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:08:35 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:08:35 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:36 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:37 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:37 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:37 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:37 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:37 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:37 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:37 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985317.84904
Apr 22 19:08:37 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:37 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:37 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:38 fir-io2-s2 kernel: ll_ost_io03_042 D ffff90fedd160000     0 85284      2 0x00000080
Apr 22 19:08:38 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:38 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:08:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:08:38 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:08:38 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:08:38 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:08:38 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:08:38 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:38 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:38 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:08:38 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:39 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:08:39 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:39 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:39 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:39 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:39 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:39 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:39 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:39 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:39 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:39 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:39 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:40 fir-io2-s2 kernel: ll_ost_io03_043 D ffff90fedd161040     0 85285      2 0x00000080
Apr 22 19:08:40 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffa90cec64>] ? __wake_up+0x44/0x50
Apr 22 19:08:40 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffa9765b71>] out_of_line_wait_on_bit_lock+0x81/0xb0
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffa9278f42>] __lock_buffer+0x32/0x40
Apr 22 19:08:41 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985321.84810
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffc09e1ea2>] do_get_write_access+0x432/0x4d0 [jbd2]
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffc15de24a>] ? ldiskfs_es_lookup_extent+0x2a/0x180 [ldiskfs]
Apr 22 19:08:41 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:08:42 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:08:42 fir-io2-s2 kernel: [<ffffffffc15b0dc0>] ldiskfs_reserve_inode_write+0x70/0xa0 [ldiskfs]
Apr 22 19:08:42 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ? ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 19:08:42 fir-io2-s2 kernel: [<ffffffffc15b0e43>] ldiskfs_mark_inode_dirty+0x53/0x210 [ldiskfs]
Apr 22 19:08:42 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 19:08:42 fir-io2-s2 kernel: [<ffffffffc16c68f1>] osd_attr_set+0x3f1/0xb90 [osd_ldiskfs]
Apr 22 19:08:42 fir-io2-s2 kernel: [<ffffffffc180c208>] ofd_commitrw_write+0x1288/0x1d40 [ofd]
Apr 22 19:08:42 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:42 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:42 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:42 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:42 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:44 fir-io2-s2 kernel: ll_ost_io00_069 D ffff90fedd162080     0 85286      2 0x00000080
Apr 22 19:08:44 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:44 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:08:44 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:08:44 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:08:44 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:08:44 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:08:44 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:08:44 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:08:44 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:08:44 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:08:44 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:08:44 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:08:45 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:08:45 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:45 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:45 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:08:45 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:08:45 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:08:45 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:45 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:45 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:45 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:45 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:45 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:46 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:47 fir-io2-s2 kernel: ll_ost_io02_065 D ffff90fedd1630c0     0 85287      2 0x00000080
Apr 22 19:08:47 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:47 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:08:47 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:08:47 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:08:47 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:08:47 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:08:47 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:08:47 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:08:47 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:08:47 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:08:47 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:08:47 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:08:47 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:48 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:49 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:49 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:49 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:49 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:49 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:49 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:49 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:49 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:49 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:49 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985329.84987
Apr 22 19:08:49 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:50 fir-io2-s2 kernel: ll_ost_io02_066 D ffff90fedd164100     0 85288      2 0x00000080
Apr 22 19:08:50 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:50 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:50 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:08:50 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:08:50 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:08:50 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:08:50 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:08:50 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:08:50 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:50 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:50 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:08:50 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:51 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:52 fir-io2-s2 kernel: ll_ost_io00_070 D ffff90fedd165140     0 85289      2 0x00000080
Apr 22 19:08:52 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:52 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:08:52 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:52 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:08:52 fir-io2-s2 kernel: [<ffffffffa9380152>] ? put_dec+0x72/0x90
Apr 22 19:08:52 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:08:52 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:08:52 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:08:52 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:08:53 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 19:08:53 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:08:53 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 19:08:53 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:08:53 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 19:08:53 fir-io2-s2 kernel: [<ffffffffc15adc26>] ? ldiskfs_getblk+0xa6/0x200 [ldiskfs]
Apr 22 19:08:53 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:08:53 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:08:53 fir-io2-s2 kernel: [<ffffffffc16e5909>] osd_ldiskfs_write_record+0xd9/0x410 [osd_ldiskfs]
Apr 22 19:08:53 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:08:53 fir-io2-s2 kernel: [<ffffffffc16e5d80>] osd_write+0x140/0x5c0 [osd_ldiskfs]
Apr 22 19:08:53 fir-io2-s2 kernel: [<ffffffffc1058116>] dt_record_write+0x36/0x120 [obdclass]
Apr 22 19:08:53 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985333.84808
Apr 22 19:08:54 fir-io2-s2 kernel: [<ffffffffc1295137>] tgt_client_data_write.isra.18+0x167/0x180 [ptlrpc]
Apr 22 19:08:54 fir-io2-s2 kernel: [<ffffffffc1297bb6>] tgt_last_rcvd_update+0x656/0xc90 [ptlrpc]
Apr 22 19:08:54 fir-io2-s2 kernel: [<ffffffffc129c9ed>] tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 19:08:54 fir-io2-s2 kernel: [<ffffffffc10593b1>] dt_txn_hook_stop+0x81/0xd0 [obdclass]
Apr 22 19:08:54 fir-io2-s2 kernel: [<ffffffffc16cc67d>] osd_trans_stop+0x18d/0x8e0 [osd_ldiskfs]
Apr 22 19:08:54 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:08:54 fir-io2-s2 kernel: [<ffffffffc16c6699>] ? osd_attr_set+0x199/0xb90 [osd_ldiskfs]
Apr 22 19:08:54 fir-io2-s2 kernel: [<ffffffffc16e759c>] ? osd_punch+0x28c/0x6b0 [osd_ldiskfs]
Apr 22 19:08:54 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:08:54 fir-io2-s2 kernel: [<ffffffffc180892b>] ofd_object_punch+0x4cb/0xd30 [ofd]
Apr 22 19:08:54 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:08:54 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:55 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:55 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:55 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:55 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:55 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:55 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:55 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:55 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:55 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:55 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:55 fir-io2-s2 kernel: ll_ost_io02_067 D ffff90fedd166180     0 85290      2 0x00000080
Apr 22 19:08:55 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:56 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:08:56 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:08:56 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:08:56 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:08:56 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:08:56 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:08:56 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:56 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:56 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:08:56 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:56 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:08:56 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:08:57 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:08:58 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:08:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:58 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:08:58 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:08:58 fir-io2-s2 kernel: ll_ost_io00_071 D ffff90fece958000     0 85291      2 0x00000080
Apr 22 19:08:58 fir-io2-s2 kernel: Call Trace:
Apr 22 19:08:58 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:08:58 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:08:58 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:08:58 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:08:58 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:08:58 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:08:58 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:08:59 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:00 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:00 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:00 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:00 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:00 fir-io2-s2 kernel: ll_ost_io02_068 D ffff90fece959040     0 85292      2 0x00000080
Apr 22 19:09:00 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:00 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:00 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:00 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:00 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:01 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:02 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:02 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:02 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:02 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:02 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:02 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:02 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:02 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:02 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:03 fir-io2-s2 kernel: ll_ost_io02_069 D ffff90fece95a080     0 85293      2 0x00000080
Apr 22 19:09:03 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:09:03 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:04 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:05 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:05 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:05 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:05 fir-io2-s2 kernel: ll_ost_io02_070 D ffff90fece95b0c0     0 85294      2 0x00000080
Apr 22 19:09:05 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:05 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:05 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:05 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:05 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:05 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:06 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:09:06 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:06 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:06 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:06 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:06 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:06 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:09:06 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:09:06 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:06 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:06 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:06 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:07 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:08 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:08 fir-io2-s2 kernel: ll_ost_io01_073 D ffff90fece95c100     0 85295      2 0x00000080
Apr 22 19:09:08 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:08 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:08 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:08 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:08 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:08 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:08 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:08 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:09:08 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:08 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:08 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:09 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:09 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:09:09 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:09 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:09 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:09 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:09 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:09 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:09 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:09 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:10 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:10 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:10 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:10 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:10 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:10 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:10 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:10 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:10 fir-io2-s2 kernel: ll_ost_io01_074 D ffff90fece95d140     0 85296      2 0x00000080
Apr 22 19:09:10 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:10 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:10 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:11 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffa90d67b0>] ? wake_up_state+0x20/0x20
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:13 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:13 fir-io2-s2 kernel: ll_ost_io02_071 D ffff90fece95e180     0 85297      2 0x00000080
Apr 22 19:09:13 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:13 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:13 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:13 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:13 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:13 fir-io2-s2 kernel: [<ffffffffa921a849>] ? ___slab_alloc+0x209/0x4f0
Apr 22 19:09:13 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:13 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:09:13 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:13 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:13 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:14 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:14 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:14 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:09:14 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:14 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:14 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:14 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:14 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:14 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:14 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:14 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:15 fir-io2-s2 kernel: ll_ost_io02_072 D ffff90ff9d04d140     0 85319      2 0x00000080
Apr 22 19:09:15 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:15 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:16 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:16 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:09:16 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:16 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:16 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985356.84769
Apr 22 19:09:16 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:16 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:16 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:09:16 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:16 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:16 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:18 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:18 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:18 fir-io2-s2 kernel: ll_ost_io02_073 D ffff90ff9d04e180     0 85320      2 0x00000080
Apr 22 19:09:18 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:18 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 19:09:18 fir-io2-s2 kernel: [<ffffffffa92aeb37>] ? dqget+0x1a7/0x450
Apr 22 19:09:18 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:09:18 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:18 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:09:18 fir-io2-s2 kernel: [<ffffffffa92afc84>] ? dquot_get_dqblk+0x144/0x1f0
Apr 22 19:09:18 fir-io2-s2 kernel: Lustre: 84927:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply#012  req@ffff90f65be3ac50 x1631545735342512/t0(0) o400->0f52cb94-f0df-9fca-8e07-7d7771e802ff@10.9.104.21@o2ib4:578/0 lens 224/0 e 1 to 0 dl 1555985363 ref 2 fl New:H/0/ffffffff rc 0/-1
Apr 22 19:09:18 fir-io2-s2 kernel: Lustre: 84927:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 49679 previous similar messages
Apr 22 19:09:19 fir-io2-s2 kernel: [<ffffffffc16f8f55>] ? osd_acct_index_lookup+0x235/0x480 [osd_ldiskfs]
Apr 22 19:09:19 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 19:09:19 fir-io2-s2 kernel: [<ffffffffa9101052>] ? ktime_get_ts64+0x52/0xf0
Apr 22 19:09:19 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:09:19 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:09:19 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:09:19 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:09:19 fir-io2-s2 kernel: [<ffffffffa9765897>] __wait_on_bit+0x67/0x90
Apr 22 19:09:19 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:09:19 fir-io2-s2 kernel: [<ffffffffa9765a01>] out_of_line_wait_on_bit+0x81/0xb0
Apr 22 19:09:19 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:09:19 fir-io2-s2 kernel: [<ffffffffc09e1cf5>] do_get_write_access+0x285/0x4d0 [jbd2]
Apr 22 19:09:20 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 19:09:20 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:09:20 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:09:20 fir-io2-s2 kernel: [<ffffffffc15b0dc0>] ldiskfs_reserve_inode_write+0x70/0xa0 [ldiskfs]
Apr 22 19:09:20 fir-io2-s2 kernel: [<ffffffffc16e3443>] ? osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:09:20 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ? ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 19:09:20 fir-io2-s2 kernel: [<ffffffffc15b0e43>] ldiskfs_mark_inode_dirty+0x53/0x210 [ldiskfs]
Apr 22 19:09:20 fir-io2-s2 kernel: [<ffffffffc15b48b0>] ldiskfs_dirty_inode+0x40/0x60 [ldiskfs]
Apr 22 19:09:20 fir-io2-s2 kernel: [<ffffffffc16e4baf>] osd_write_commit+0x41f/0x8c0 [osd_ldiskfs]
Apr 22 19:09:20 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:21 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:22 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:22 fir-io2-s2 kernel: ll_ost_io02_074 D ffff90ff9d049040     0 85321      2 0x00000080
Apr 22 19:09:22 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:22 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:22 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:22 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:22 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:22 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:22 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 19:09:23 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:24 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:24 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:24 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:24 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:24 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:24 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:24 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:24 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:24 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:24 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:24 fir-io2-s2 kernel: ll_ost_io00_072 D ffff90ff9d04a080     0 85322      2 0x00000080
Apr 22 19:09:24 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:24 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:25 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:25 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:25 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:25 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:25 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:09:25 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:25 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:25 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:25 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:25 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:25 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:09:25 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:26 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:27 fir-io2-s2 kernel: ll_ost_io03_044 D ffff90ff9d04b0c0     0 85323      2 0x00000080
Apr 22 19:09:27 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:27 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:27 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:27 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:27 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:27 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:27 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 19:09:27 fir-io2-s2 kernel: [<ffffffffc09e0cae>] jbd2__journal_restart+0xfe/0x160 [jbd2]
Apr 22 19:09:28 fir-io2-s2 kernel: [<ffffffffc09e0d23>] jbd2_journal_restart+0x13/0x20 [jbd2]
Apr 22 19:09:28 fir-io2-s2 kernel: [<ffffffffc15ad25e>] ldiskfs_truncate_restart_trans+0x4e/0x80 [ldiskfs]
Apr 22 19:09:28 fir-io2-s2 kernel: [<ffffffffc15ec132>] ldiskfs_ext_truncate_extend_restart+0x42/0x60 [ldiskfs]
Apr 22 19:09:28 fir-io2-s2 kernel: [<ffffffffc15efb3d>] ldiskfs_ext_remove_space+0x57d/0x1110 [ldiskfs]
Apr 22 19:09:28 fir-io2-s2 kernel: [<ffffffffa921b9d2>] ? kmem_cache_free+0x1e2/0x200
Apr 22 19:09:28 fir-io2-s2 kernel: [<ffffffffc15dd2a9>] ? ldiskfs_es_free_extent+0x59/0x60 [ldiskfs]
Apr 22 19:09:28 fir-io2-s2 kernel: [<ffffffffc15f2504>] ? ldiskfs_ext_truncate+0x24/0xe0 [ldiskfs]
Apr 22 19:09:28 fir-io2-s2 kernel: [<ffffffffc15b0d38>] ? ldiskfs_mark_iloc_dirty+0x68/0x80 [ldiskfs]
Apr 22 19:09:28 fir-io2-s2 kernel: [<ffffffffc15f2590>] ldiskfs_ext_truncate+0xb0/0xe0 [ldiskfs]
Apr 22 19:09:28 fir-io2-s2 kernel: [<ffffffffc15b26c7>] ldiskfs_truncate+0x3b7/0x3f0 [ldiskfs]
Apr 22 19:09:28 fir-io2-s2 kernel: [<ffffffffc129c9ed>] ? tgt_txn_stop_cb+0x19d/0x490 [ptlrpc]
Apr 22 19:09:28 fir-io2-s2 kernel: [<ffffffffc16e7280>] osd_execute_truncate+0xe0/0x170 [osd_ldiskfs]
Apr 22 19:09:29 fir-io2-s2 kernel: [<ffffffffc16e7a0d>] osd_process_truncates+0x4d/0x90 [osd_ldiskfs]
Apr 22 19:09:29 fir-io2-s2 kernel: [<ffffffffc16cc6ee>] osd_trans_stop+0x1fe/0x8e0 [osd_ldiskfs]
Apr 22 19:09:29 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:09:29 fir-io2-s2 kernel: [<ffffffffc16c6699>] ? osd_attr_set+0x199/0xb90 [osd_ldiskfs]
Apr 22 19:09:29 fir-io2-s2 kernel: [<ffffffffc16e759c>] ? osd_punch+0x28c/0x6b0 [osd_ldiskfs]
Apr 22 19:09:29 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:09:29 fir-io2-s2 kernel: [<ffffffffc180892b>] ofd_object_punch+0x4cb/0xd30 [ofd]
Apr 22 19:09:29 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:09:29 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:29 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:29 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:29 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:30 fir-io2-s2 kernel: ll_ost_io02_075 D ffff90ff9d04c100     0 85324      2 0x00000080
Apr 22 19:09:30 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:30 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:31 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:32 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:32 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:32 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:32 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:32 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:32 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:32 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:32 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:32 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:32 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:32 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:33 fir-io2-s2 kernel: ll_ost_io00_073 D ffff90fc7cf79040     0 85325      2 0x00000080
Apr 22 19:09:33 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:33 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:09:34 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:09:34 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:34 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:34 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:34 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:34 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:34 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:34 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:34 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:34 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:34 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:34 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:35 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:35 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:35 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:35 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:35 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:35 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:35 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:35 fir-io2-s2 kernel: ll_ost_io01_075 D ffff90fc7cf7d140     0 85326      2 0x00000080
Apr 22 19:09:35 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:35 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:35 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:35 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:36 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:37 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:37 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:37 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:37 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:37 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:37 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:37 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:37 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:37 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:37 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:37 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:37 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:38 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:38 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:38 fir-io2-s2 kernel: ll_ost_io03_045 D ffff90fc7cf78000     0 85327      2 0x00000080
Apr 22 19:09:38 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:38 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:38 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:38 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:38 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:38 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:38 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:09:38 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:38 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:38 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:39 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:39 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:39 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:09:39 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:09:39 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:39 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:39 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:39 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:39 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:39 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:39 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:39 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:40 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:40 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:40 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:40 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:40 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:40 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:40 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:40 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:40 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:40 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:40 fir-io2-s2 kernel: ll_ost_io00_074 D ffff90fc7cf7e180     0 85328      2 0x00000080
Apr 22 19:09:40 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:40 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffa92b162f>] __dquot_alloc_space+0x1ef/0x250
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffa93a01c1>] ? __percpu_counter_add+0x51/0x70
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffc15e5941>] ldiskfs_mb_new_blocks+0xe1/0xb20 [ldiskfs]
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffc16ba728>] ? osd_read_unlock+0x58/0x90 [osd_ldiskfs]
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffc16615a2>] ? lquota_disk_read+0xf2/0x390 [lquota]
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffc15ed742>] ? ldiskfs_ext_find_extent+0x292/0x340 [ldiskfs]
Apr 22 19:09:41 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:09:42 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:09:42 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:09:42 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:09:42 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:09:42 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:09:42 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:09:42 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:42 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:42 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:42 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:42 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:42 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:43 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:43 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:43 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:43 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:43 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:43 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:43 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:43 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:43 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:43 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:43 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:43 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:44 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:44 fir-io2-s2 kernel: ll_ost_io01_076 D ffff90fc7cf7c100     0 85329      2 0x00000080
Apr 22 19:09:44 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:44 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:44 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:09:44 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:09:44 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:44 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:09:44 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:09:44 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:44 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:44 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:44 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:44 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:45 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:46 fir-io2-s2 kernel: ll_ost_io01_077 D ffff90fc7cf7b0c0     0 85330      2 0x00000080
Apr 22 19:09:46 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:46 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:46 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:46 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:46 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:46 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:46 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:09:46 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:46 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:46 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:46 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:46 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:47 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:09:47 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:09:47 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:47 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:47 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:47 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:47 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:47 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:47 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:47 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:47 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:47 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:48 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:48 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:48 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:48 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:48 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:48 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:48 fir-io2-s2 kernel: ll_ost_io03_046 D ffff90fc7cf7a080     0 85331      2 0x00000080
Apr 22 19:09:48 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:48 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:48 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:09:48 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:48 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:09:48 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:09:49 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:09:49 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:49 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:09:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:49 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:09:49 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:09:49 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:09:49 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:09:49 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:49 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:49 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:49 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:50 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:51 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:51 fir-io2-s2 kernel: ll_ost_io00_075 D ffff90fc7cf5a080     0 85332      2 0x00000080
Apr 22 19:09:51 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:51 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:51 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:09:51 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:09:51 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:51 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:09:51 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:09:51 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:51 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:51 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:51 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffa9765e92>] ? mutex_lock+0x12/0x2f
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:52 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:53 fir-io2-s2 kernel: ll_ost_io01_078 D ffff90fc7cf58000     0 85333      2 0x00000080
Apr 22 19:09:53 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:53 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:53 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:09:53 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:09:53 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:09:53 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:09:53 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:09:53 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:53 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:53 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:53 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:53 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:54 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:55 fir-io2-s2 kernel: ll_ost_io01_079 D ffff90fc7cf5e180     0 85334      2 0x00000080
Apr 22 19:09:55 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa9765721>] schedule_timeout+0x221/0x2d0
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa90c2cc5>] ? wake_up_bit+0x25/0x30
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa97672ed>] io_schedule_timeout+0xad/0x130
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa9767388>] io_schedule+0x18/0x20
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa9765d71>] bit_wait_io+0x11/0x50
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa9765921>] __wait_on_bit_lock+0x61/0xc0
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa9765d60>] ? bit_wait+0x50/0x50
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa9765b71>] out_of_line_wait_on_bit_lock+0x81/0xb0
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa90c2dc0>] ? wake_bit_function+0x40/0x40
Apr 22 19:09:55 fir-io2-s2 kernel: [<ffffffffa9278f42>] __lock_buffer+0x32/0x40
Apr 22 19:09:56 fir-io2-s2 kernel: [<ffffffffc09e1ea2>] do_get_write_access+0x432/0x4d0 [jbd2]
Apr 22 19:09:56 fir-io2-s2 kernel: [<ffffffffa927918d>] ? __getblk+0x2d/0x300
Apr 22 19:09:56 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:09:56 fir-io2-s2 kernel: [<ffffffffc09ec635>] ? jbd2_journal_add_journal_head+0x65/0x150 [jbd2]
Apr 22 19:09:56 fir-io2-s2 kernel: [<ffffffffc09e1f67>] jbd2_journal_get_write_access+0x27/0x40 [jbd2]
Apr 22 19:09:56 fir-io2-s2 kernel: [<ffffffffc15d351b>] __ldiskfs_journal_get_write_access+0x3b/0x80 [ldiskfs]
Apr 22 19:09:56 fir-io2-s2 kernel: [<ffffffffc15e1cf8>] ldiskfs_mb_mark_diskspace_used+0xf8/0x510 [ldiskfs]
Apr 22 19:09:56 fir-io2-s2 kernel: [<ffffffffc15e5bb0>] ldiskfs_mb_new_blocks+0x350/0xb20 [ldiskfs]
Apr 22 19:09:56 fir-io2-s2 kernel: [<ffffffffc15ec545>] ? __read_extent_tree_block+0x55/0x1e0 [ldiskfs]
Apr 22 19:09:56 fir-io2-s2 kernel: [<ffffffffa921d61b>] ? __kmalloc+0x1eb/0x230
Apr 22 19:09:56 fir-io2-s2 kernel: [<ffffffffc15ed5e4>] ? ldiskfs_ext_find_extent+0x134/0x340 [ldiskfs]
Apr 22 19:09:56 fir-io2-s2 kernel: [<ffffffffc1675c7b>] ? qsd_acquire+0x19b/0xcb0 [lquota]
Apr 22 19:09:57 fir-io2-s2 kernel: [<ffffffffc15f1a26>] ldiskfs_ext_map_blocks+0x4a6/0xf60 [ldiskfs]
Apr 22 19:09:57 fir-io2-s2 kernel: [<ffffffffc15ad5d5>] ldiskfs_map_blocks+0x155/0x700 [ldiskfs]
Apr 22 19:09:57 fir-io2-s2 kernel: [<ffffffffc16e3443>] osd_ldiskfs_map_inode_pages+0x143/0x420 [osd_ldiskfs]
Apr 22 19:09:57 fir-io2-s2 kernel: [<ffffffffc16e4b32>] osd_write_commit+0x3a2/0x8c0 [osd_ldiskfs]
Apr 22 19:09:57 fir-io2-s2 kernel: LNet: Service thread pid 83793 was inactive for 753.99s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Apr 22 19:09:57 fir-io2-s2 kernel: LNet: Skipped 4 previous similar messages
Apr 22 19:09:57 fir-io2-s2 kernel: Pid: 83793, comm: ll_ost00_029 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 19:09:57 fir-io2-s2 kernel: Call Trace:
Apr 22 19:09:57 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:09:57 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:09:57 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:57 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:57 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:57 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:09:57 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 19:09:57 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985397.83793
Apr 22 19:09:58 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:09:58 fir-io2-s2 kernel: [<ffffffffc180bf3e>] ofd_commitrw_write+0xfbe/0x1d40 [ofd]
Apr 22 19:09:58 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:09:58 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:09:58 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:09:58 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:09:58 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:09:58 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:09:59 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:09:59 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:09:59 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:09:59 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:09:59 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:09:59 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:09:59 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:09:59 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:09:59 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:09:59 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:09:59 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:09:59 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:00 fir-io2-s2 kernel: ll_ost_io00_076 D ffff90fc7cf5b0c0     0 85335      2 0x00000080
Apr 22 19:10:00 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffc16fa3d0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:10:00 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:01 fir-io2-s2 kernel: [<ffffffffc180be9e>] ofd_commitrw_write+0xf1e/0x1d40 [ofd]
Apr 22 19:10:01 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:10:01 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:10:01 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:10:01 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:10:01 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:10:01 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:10:01 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:10:01 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:10:01 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:10:01 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:10:01 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:10:02 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:10:02 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:10:02 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:10:02 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:10:02 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:10:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:02 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:02 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:02 fir-io2-s2 kernel: ll_ost_io01_080 D ffff90fc7cf5c100     0 85336      2 0x00000080
Apr 22 19:10:02 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:02 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:10:02 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:10:02 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:10:03 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:10:04 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:10:04 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:10:04 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:10:04 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:10:04 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:10:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:04 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:04 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:04 fir-io2-s2 kernel: ll_ost_io03_047 D ffff90fc7cf5d140     0 85337      2 0x00000080
Apr 22 19:10:04 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:04 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:10:04 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 19:10:04 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:10:05 fir-io2-s2 kernel: Pid: 84825, comm: ll_ost00_051 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 19:10:05 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc15caac3>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffa92aed8a>] dqget+0x3fa/0x450
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffa92afb54>] dquot_get_dqblk+0x14/0x1f0
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc16f8f55>] osd_acct_index_lookup+0x235/0x480 [osd_ldiskfs]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc165e80d>] lquotactl_slv+0x27d/0x9d0 [lquota]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc17f1a3c>] ofd_quotactl+0x13c/0x380 [ofd]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 19:10:05 fir-io2-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1555985405.84825
Apr 22 19:10:05 fir-io2-s2 kernel: Pid: 83767, comm: ll_ost00_021 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018
Apr 22 19:10:05 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc129e732>] req_can_reconstruct+0x42/0x100 [ptlrpc]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc12a73da>] tgt_request_handle+0xdba/0x1580 [ptlrpc]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:05 fir-io2-s2 kernel: [<ffffffffffffffff>] 0xffffffffffffffff
Apr 22 19:10:07 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:10:07 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:10:08 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:10:09 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:10:09 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:10:09 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:10:09 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:10:09 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:10:09 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:10:09 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:10:09 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:10:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:09 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:09 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:10 fir-io2-s2 kernel: ll_ost_io03_048 D ffff90fc7cf59040     0 85338      2 0x00000080
Apr 22 19:10:10 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:10 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:10:10 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:10:10 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:10:10 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:10:10 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:10:10 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:10:10 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:10:10 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:10:10 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:10 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:10:10 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:10 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:10:11 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:10:12 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:10:12 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:10:12 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:10:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:12 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:12 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:12 fir-io2-s2 kernel: ll_ost_io03_049 D ffff90fecae18000     0 85339      2 0x00000080
Apr 22 19:10:12 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:12 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:10:12 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:10:12 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:10:12 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:10:12 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:10:13 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:10:13 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:10:13 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:10:13 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:10:13 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:10:13 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:10:13 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:10:13 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:10:13 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:13 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:10:13 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:10:13 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:10:14 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:10:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:15 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:15 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:15 fir-io2-s2 kernel: ll_ost_io03_050 D ffff90fecae19040     0 85340      2 0x00000080
Apr 22 19:10:15 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:15 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:10:15 fir-io2-s2 kernel: [<ffffffffc16cc755>] osd_trans_stop+0x265/0x8e0 [osd_ldiskfs]
Apr 22 19:10:15 fir-io2-s2 kernel: [<ffffffffc15b48c4>] ? ldiskfs_dirty_inode+0x54/0x60 [ldiskfs]
Apr 22 19:10:15 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:10:15 fir-io2-s2 kernel: [<ffffffffc1804e65>] ofd_trans_stop+0x25/0x60 [ofd]
Apr 22 19:10:15 fir-io2-s2 kernel: [<ffffffffc180b954>] ofd_commitrw_write+0x9d4/0x1d40 [ofd]
Apr 22 19:10:15 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:10:15 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:10:15 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:10:16 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:10:16 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:10:16 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:10:16 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:10:16 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:10:16 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:10:16 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:10:16 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:10:16 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:10:16 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:10:16 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:10:16 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:10:17 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:10:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:17 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:17 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:17 fir-io2-s2 kernel: ll_ost_io03_051 D ffff90fecae1a080     0 85341      2 0x00000080
Apr 22 19:10:17 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:17 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:10:17 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:10:17 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:10:17 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:10:17 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:10:17 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:10:17 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:10:18 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:10:19 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:10:19 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:10:19 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:10:19 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:10:19 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:10:19 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:10:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:19 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:19 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:19 fir-io2-s2 kernel: ll_ost_io01_081 D ffff90fecae1b0c0     0 85342      2 0x00000080
Apr 22 19:10:19 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:19 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:10:19 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:10:19 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:10:20 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:10:20 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:10:20 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:10:20 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:10:20 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:10:20 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:20 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:10:20 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:20 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:10:20 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:10:20 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:10:20 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:10:21 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:10:21 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:10:21 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:10:21 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:10:21 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:10:21 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:10:21 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:10:21 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:10:21 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:10:21 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:10:21 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:10:21 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:10:22 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:10:22 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:10:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:22 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:22 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:22 fir-io2-s2 kernel: ll_ost_io03_052 D ffff90fecae1c100     0 85343      2 0x00000080
Apr 22 19:10:22 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:22 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:10:22 fir-io2-s2 kernel: [<ffffffffc0537b30>] ? _mlx5_ib_post_send+0x3a0/0x13c0 [mlx5_ib]
Apr 22 19:10:22 fir-io2-s2 kernel: [<ffffffffa9768b69>] schedule_preempt_disabled+0x29/0x70
Apr 22 19:10:22 fir-io2-s2 kernel: [<ffffffffa9766ab7>] __mutex_lock_slowpath+0xc7/0x1d0
Apr 22 19:10:22 fir-io2-s2 kernel: [<ffffffffa9765e9f>] mutex_lock+0x1f/0x2f
Apr 22 19:10:22 fir-io2-s2 kernel: [<ffffffffa92ae2d9>] dquot_commit+0x29/0xd0
Apr 22 19:10:22 fir-io2-s2 kernel: [<ffffffffc15cab8c>] ldiskfs_write_dquot+0x6c/0x90 [ldiskfs]
Apr 22 19:10:23 fir-io2-s2 kernel: [<ffffffffc15cabef>] ldiskfs_mark_dquot_dirty+0x3f/0x60 [ldiskfs]
Apr 22 19:10:23 fir-io2-s2 kernel: [<ffffffffa92b106a>] __dquot_transfer+0x24a/0x540
Apr 22 19:10:23 fir-io2-s2 kernel: [<ffffffffc15b637d>] ? ldiskfs_xattr_get+0x7d/0x2f0 [ldiskfs]
Apr 22 19:10:23 fir-io2-s2 kernel: [<ffffffffa9760000>] ? load_elf_interp.constprop.11+0xd1/0x426
Apr 22 19:10:23 fir-io2-s2 kernel: [<ffffffffa92a0000>] ? locks_remove_file+0xe0/0x210
Apr 22 19:10:23 fir-io2-s2 kernel: [<ffffffffa92b13dc>] dquot_transfer+0x7c/0xe0
Apr 22 19:10:23 fir-io2-s2 kernel: [<ffffffffc16b948d>] osd_quota_transfer+0xdd/0x210 [osd_ldiskfs]
Apr 22 19:10:23 fir-io2-s2 kernel: [<ffffffffc09e0a93>] ? jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:10:23 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:23 fir-io2-s2 kernel: [<ffffffffc16c661f>] osd_attr_set+0x11f/0xb90 [osd_ldiskfs]
Apr 22 19:10:23 fir-io2-s2 kernel: [<ffffffffc16c6255>] ? osd_trans_start+0x235/0x4e0 [osd_ldiskfs]
Apr 22 19:10:23 fir-io2-s2 kernel: [<ffffffffc180c342>] ofd_commitrw_write+0x13c2/0x1d40 [ofd]
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:10:24 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:10:25 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:10:25 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:10:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:25 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:25 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:25 fir-io2-s2 kernel: ll_ost_io00_077 D ffff90fecae1d140     0 85353      2 0x00000080
Apr 22 19:10:25 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:25 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:10:25 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:10:25 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:10:25 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:10:25 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:10:25 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:10:25 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:10:26 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:10:26 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:26 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:10:26 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:26 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:10:26 fir-io2-s2 kernel: [<ffffffffc180b9b1>] ofd_commitrw_write+0xa31/0x1d40 [ofd]
Apr 22 19:10:26 fir-io2-s2 kernel: [<ffffffffc180fdc2>] ofd_commitrw+0x4b2/0xa10 [ofd]
Apr 22 19:10:26 fir-io2-s2 kernel: [<ffffffffc12a0bdc>] obd_commitrw+0x9c/0x370 [ptlrpc]
Apr 22 19:10:26 fir-io2-s2 kernel: [<ffffffffc12a384d>] tgt_brw_write+0x100d/0x1a90 [ptlrpc]
Apr 22 19:10:26 fir-io2-s2 kernel: [<ffffffffc0bf4a58>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
Apr 22 19:10:26 fir-io2-s2 kernel: [<ffffffffa90e011c>] ? update_curr+0x14c/0x1e0
Apr 22 19:10:26 fir-io2-s2 kernel: [<ffffffffa90dca7e>] ? account_entity_dequeue+0xae/0xd0
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffc11f8890>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffc1248565>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffa90d67c2>] ? default_wake_function+0x12/0x20
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffa90cba9b>] ? __wake_up_common+0x5b/0x90
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffc124f00c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc]
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffc124e510>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffa90c1c31>] kthread+0xd1/0xe0
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffa90d0884>] ? finish_task_switch+0x54/0x1c0
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffa9774c24>] ret_from_fork_nospec_begin+0xe/0x21
Apr 22 19:10:27 fir-io2-s2 kernel: [<ffffffffa90c1b60>] ? insert_kthread_work+0x40/0x40
Apr 22 19:10:28 fir-io2-s2 kernel: ll_ost_io00_078 D ffff90fecae1e180     0 85354      2 0x00000080
Apr 22 19:10:28 fir-io2-s2 kernel: Call Trace:
Apr 22 19:10:28 fir-io2-s2 kernel: [<ffffffffa9767c49>] schedule+0x29/0x70
Apr 22 19:10:28 fir-io2-s2 kernel: [<ffffffffc09e0085>] wait_transaction_locked+0x85/0xd0 [jbd2]
Apr 22 19:10:28 fir-io2-s2 kernel: [<ffffffffa90c2d00>] ? wake_up_atomic_t+0x30/0x30
Apr 22 19:10:28 fir-io2-s2 kernel: [<ffffffffc09e0368>] add_transaction_credits+0x268/0x2f0 [jbd2]
Apr 22 19:10:28 fir-io2-s2 kernel: [<ffffffffc09e05e1>] start_this_handle+0x1a1/0x430 [jbd2]
Apr 22 19:10:28 fir-io2-s2 kernel: [<ffffffffc16be8b1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
Apr 22 19:10:28 fir-io2-s2 kernel: [<ffffffffa921c0b2>] ? kmem_cache_alloc+0x1c2/0x1f0
Apr 22 19:10:28 fir-io2-s2 kernel: [<ffffffffc09e0a93>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
Apr 22 19:10:28 fir-io2-s2 kernel: [<ffffffffc16c622e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:28 fir-io2-s2 kernel: [<ffffffffc15d32c9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
Apr 22 19:10:28 fir-io2-s2 kernel: [<ffffffffc16c622e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
Apr 22 19:10:29 fir-io2-s2 kernel: [<ffffffffc1804dc5>] ofd_trans_start+0x75/0xf0 [ofd]
Apr 22 19:10:29 fir-io2-s2 kernel: [<ffffffffc1808baa>] ofd_object_punch+0x74a/0xd30 [ofd]
Apr 22 19:10:29 fir-io2-s2 kernel: [<ffffffffc17f5cf3>] ofd_punch_hdl+0x493/0xa30 [ofd]
Apr 22 19:10:29 fir-io2-s2 kernel: [<ffffffffc12a710a>] tgt_request_handle+0xaea/0x1580 [ptlrpc]
Apr 22 19:10:29 fir-io2-s2 kernel: [<ffffffffc0bf4f07>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
Apr 22 19:10:29 fir-io2-s2 kernel: [<ffffffffc124b6db>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]